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MANAGEMENT  SUMMARY 


The  Department  of  Defense  (DoD)  perceives  software  as  an 
increasing  cost  which  must  be  harnessed  and  controlled.  This 
perception  was  first  evident  in  the  mid- 1970 's  when  DoD  Direc¬ 
tive  5000.29,  Management  of  Computer  Resources  in  Major  Defense 
Systems,  and  DoD  Instruction  5000.31,  Interim  List  of  DoD  Ap¬ 
proved  High  Order  Programming  Languages,  were  issued.  Another 
related  undertaking  in  the  later  1970' s  and  still  in  the  Research 
5  Development  stages  is  an  effort  to  standardize  on  computer 
architectures  with  the  intent  of  reducing  costs.  Recently,  a  new 
DoD  Software  Technology  Initiative  has  been  assembled  aimed  at 
order-of -magnitude  improvements  in  DoD's  capabilities  to  develop 
and  maintain  software.  The  need  for  this  is  evident  because  the 
cost  for  software  within  DoD  is  projected  to  grow  from  $3  billion/ 
year  currently  to  $30  billion/year  over  the  next  decade. 

The  services  responded  to  DoD  direction  by  issuance  of 
regulations,  standards,  and  specifications,  aimed  at  controlling 
software  development.  The  responses  were,  for  the  most  part,  on 
an  individual,  service  and  agency  basis  and  resulted  in  a  pro¬ 
liferation  of  software  acquisition  guidelines,  management  pro¬ 
cedures  and  standardizat ion  efforts.  This  caused  some  overlap 
and  redundancy.  To  rectify  this,  a  Joint  Logistics  Commanders 
(JLC)  Software  Workshop  was  convened  from  April  2  to  April  4,  1979. 
Four  panels  were  set  up:  (1)  Panel  A,  Software  Acquisition/ 
Development  Standards,  (2)  Panel  B,  Software  Documentation,  (3) 
Panel  C,  Standards  for  Software  Quality,  and  (4)  Panel  D,  Software 
Acceptance  Criteria.  The  panels  issued  findings  and  recommen¬ 
dations.  The  JLC  Joint  Policy  Coordinating  Group  for  Computer 
Resource  Management  then  asked  the  Computer  Software  Management 
(CSM)  Subgroup  to  develop  a  plan  of  action  for  solving  the 
problems  identified  by  the  four  (4)  panels  above.  This  plan  is 
being  implemented. 


This  effort,  "Improving  Methods  of  Assuring  Quality  Software", 
was  conceived  as  a  result  of  the  JLC  study  and  is  related  to  the  > 

CSM  plan  of  action.  It  was  a  co-ordinated  effort  among  Rome  Air 
Development  Center  (RADC) ,  Defense  Logistics  Agency  Headquarters 
(DLA  Hq) ,  Air  Force  Contracts  Management  Division  Headquarters 
(AFCMD  Hq)  and  Electronic  Systems  Division  (ESD) .  RADC  contracted 
with  Systems  Architects,  Inc.  (SAI)  to  perform  the  effort.  The 
findings  will  be  forwarded  to  the  Air  Force  representative  for 
consideration  in  the  JLC  program.  V 

Systems  Architects,  Inc.  (SAI)  has  examined,  analyzed  and 
evaluated  the  current  software  acquisition  and  contract  admini-  > 
stration  management  documents,  software  quality  assurance  tools, 
techniques  and  communication  methods  and  has  developed  a  series 
of  recommendations  for  improved  methods  of  assuring  quality  soft¬ 
ware.  These  improved  methods  encompass  the  entire  software  devel¬ 
opment  life  cycle  which  consists  of  five  phases:  (1)  Require-' 
ment  Analysis,  (2)  Design,  (3)  Code  and  Checkout,  (4)  Test  and 
Integration,  (5)  Operation  and  Maintenance.  SAI  examined  relevant 
documentation,  conducted  interviews  and  compiled  the  results  from 
a  comprehensive  questionnaire  as  the  basis  for  the  analysis, 
evaluation  and  recommendations  which  can  be  found  herein. 

SAI's  recommendations  for  improved  methods  of  assuring  quality 
software  are  classified  in  four  classes:  (1)  Establish  clear, 
unambiguous  Government  Software  Quality  Assurance  Guidance  Docu¬ 
ments,  (2)  Include  Software  Quality  Assurance  Functions  in  all 
phases  of  the  Software  Development  Life  Cycle,  (3)  Improve  communi¬ 
cation  methods  and  model  documents  primarily  by  mutual  agreement 
regarding  allocation  of  functional  responsibilities  between  CAO's 
and  Program  Offices,  and  (4)  Provide  up  to  date  training  and  people 
skilled  in  software  development  to  government  SQA  organizations. 

Establishing  clear,  unambiguous  government  software  quality 
assurance  plans  is  a  prerequisite  for  performing  the  software 
quality  assurance  function.  Most  current  plans  have  evolved  out 
of  the  hardware  quality  control  area.  The  adaptation  of  a  hard- 


ware  quality  control  plan  to  a  software  quality  assurance  plan 
does  not  meet  all  requirements  due  to  the  two  fundamentally 
different  natures  of  hardware  and  software.  Hardware  quality 
control  is  oriented  toward  the  production  and  manufacturing  phase 
of  a  reproduceable  product.  In  software  quality  assurance,  there 
is  a  need  for  involvement  during  the  entire  software  development 
life  cycle  since  'the  product  can  be  considered  to  be  produced 
only  one  time.  Software  quality  assurance  is  a  separate  issue 
and  a  unique  discipline  and  therefore  requires  separate  unique 
guidance  documents. 

Including  the  software  quality  assurance  function  within 
all  phases  of  the  software  development  life  cycle  is  essential  to 
the  development  of  quality  software.  The  government's  SQA  repre¬ 
sentatives  should  monitor  the  implementation  of  the  contractor's 
SQA  plan  to  ensure  all  SQA  functions  are  covered  during  the 
appropriate  phases  of  the  software  development  life  cycle.  During 
the  Requirements  Analysis  Phase,  the  government  SQA  organization 
should  assist  in  the  review  of  the  requirements  specification  to 
help  analyze  and  evaluate  the  software  requirements  for  accept¬ 
ability.  During  the  Design  Phase,  the  government  SQA  organization 
should  work  with  the  software  development  team  to  recommend  and 
then  maintain  standards,  procedures,  and  plans  affecting  the 
balance  of  the  development  process.  During  the  Code  and  Checkout 
Phase,  numerous  SQA  monitoring  functions  are  conducted  to  verify 
that  development  personnel  have  implemented  all  requirements  and 
complied  with  project  standards.  During  Testing  and  Evaluation, 
the  SQA  organization  should  review  project  test  plans  and  pro¬ 
cedures.  The  review  should  trace  requirements  from  the  specifi¬ 
cation  to  the  test  plans  and  test  reports.  During  Operation  and 
Maintenance,  the  SQA  organization  should  maintain  software  trend 
analysis  studies  to  measure  the  frequency  of  defects  as  an  aid  to 
the  maintenance  group. 

Communication  methods  within  and  among  SPOs  and  CAOs  should 


be  improved.  Currently  ESD,  DLA  and  AFCMD  ail  operate  on  different 
vocabularies.  This  difference  has  been  recognized  at  Air  Force 
Headquarters  level,  and  ESD  and  AFCMD  are  currently  working  toward 
closer  models  and  vocabularies.  Delegation  of  functions  for  SQA 
by  the  Program  Office  through  Memoranda  of  Agreement  (MOAs)  and 
Letters  of  Delegation  (LODs)  sometimes  causes  problems  because 
the  CAO  does  not  have  the  resources  to  perform  a  function.  The 
SPO  and  CAO  need  to  "communicate"  to  cover  all  functions  before 
the  formal  agreement  of  delegation  of  functions  before  either  a 
MOA  or  LOD  is  written.  This  communication  should  continue  when 
necessary  throughout  the  contract. 

Providing  up  to  date  training  and  people  knowledgeable  in 
software  development  to  government  SQA  organizations  is  a  must  if 
the  above  three  recommendations  are  to  be  carried  through  to 
implementation.  A  member  of  a  SQA  organization  representing  the 
government  to  a  software  development  contractor  will  provide  his 
important  service  to  the  government  only  if  he  has  the  techno¬ 
logical  "know-how".  Government  SQA  personnel  must  have  knowledge 
in  software  development  to  be  effective. 

This  required  knowledge  for  government  SQA  organizations  can 
be  gained  either  through  training  of  the  current  staff  or  hiring 
from  the  outside.  It  should  be  recognized  that  in  competing  for 
this  knowledge  and  skill,  the  government  is  competing  for  scarce 
personnel.  As  part  of  this  effort,  SAI  has  prepared  and  delivered 
a  "Training  Outline"  customized  for  POs'  and  CAOs'  training  needs 
under  separate  cover. 

This  effort  should  be  viewed  as  the  catalyst  for  continuing 
endeavors  by  ESD,  AFCMD  and  DLA  in  their  necessary  commitment  to 
SQA. 

Again,  the  projected  cost  for  software  in  1990  within  DoD  is 
$30  billion.  Many  of  the  programs  being  monitored  in  the  field 
are  just  now  experiencing  contemporary  software  system  develop- 
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ment  practices.  New  practices  evolving  from  the  R  §  D  community 
are  on  the  horizon  and  others  can  be  expected  in  this  rapidly 
changing  and  dynamic  field.  The  DoD  Software  Technology  Initiative 
should  serve  as  a  motivating  factor. 

There  are  reports  that  users  tasked  with  the  maintenance  of 
operational  software  are  saturated  to  the  point  where  they  cannot 
see  how  they  will  be  able  to  accept  and  maintain  new  software. 
Software  maintenance  costs  can  be  expected  to  continue  to  be  the 
largest  part  of  the  $30  billion.  Effective  SQA  will  reduce  this 
cost . 

Adequate  and  modular  training  programs,  opportunity  for  per¬ 
sonnel  advancement,  better  planning,  and  communication  between  POs 
and  CAOs ,  are  some  of  the  necessary  ingredients  for  insuring  the 
delivery  of  high  quality  software  and  alleviating  what  will  other¬ 
wise  be  an  unmanageable  maintenance  problem. 

Implementation  of  these  recommendations  will  provide  DoD  with 
an  increased  probability  of  receiving  and  maintaining  quality  soft¬ 
ware  . 
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SECTION  I 


INTRODUCTION  AND  WORKPLAN 


1.1  INTRODUCTION 

The  Electronic  Systems  Division  (ESD)  Project  Offices 
(POs)  Hanscom  AFB,  MA;  the  Air  Force  Contract  Management  Divi¬ 
sion  (AFCMD) ,  Kirtland  AFB,  NM ;  and  its  field  Air  Force  Plant 
Representative  Offices  (AFPROs) ;  and  the  Defense  Logistics 
Agency  (DLA),  Alexandria,  VA;  and  its  field  offices;  Defense 
Contract  Administration  Services  Plant  Representative  Offices 
(DCASPROs)  and  Defense  Contract  Administration  Services  Manage¬ 
ment  Areas  (DCASMAs)  have  an  increased  demand  for  tools,  tech¬ 
niques  and  methods  to  ensure  that  the  Government  is  receiving 
acceptable  software.  The  current  software  quality  tools,  tech¬ 
niques  and  methods  are  in  need  of  improvement.  In  order  to 
improve  the  current  software  quality  tools,  techniques  and 
methods,  the  Rome  Air  Development  Center  (RADC)  contracted 
with  Systems  Architects,  Inc.  (SAI)  to  perform  an  analysis  and 
evaluation  of  current  software  quality  tools,  techniques  and 
methods  and  make  recommendations  for  their  improvement.  The 
Statement  of  Work  for  this  contract  was  co-ordinated  among 
RADC,  DLA  Hq.,  AFCMD  Hq . ,  and  ESD/TOIQ,  recently  redesignated 
ESD/TOEA . 

1.2  OBJECTIVE 

The  objective  of  this  study  was  to  analyze,  evaluate  and 
improve  the  software  quality  tools,  techniques  and  methods  us>ed 
by  ESD,  DLA  and  AFCMD.  These  improved  software  quality  tools, 
techniques  and  methodologies  will  enable  the  ESD  Project  Offices 
to  provide  more  effective  guidance  to  the  acquisition  manager  of 
contracts  for  software  development;  enhance  the  ability  of  the 
ESD  Project  Offices  and  Contract  Administration  Offices  (CAOs) 
(Hq .  of  AFCMD  and  its  AFPROs  and  Hq.  DLA  and  its  subordinate 
organizations)  to  monitor  the  progress  of  software  quality 
during  the  different  phases  of  the  software  development  life 
cycle;  and  better  the  Government  probability  of  receiving 
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software  of  acceptable  quality  from  defense  contractors. 
1.3  SAI's  WORKPLAN 


SAI  performed  this  study  through  the  completion  of  the 
following  six  tasks  as  scheduled:  (Task  I)  Project  Kickoff 
and  Background  Research,  (Task  II)  Structured  Data  Collection 
Instruments  and  On-Site  Interviews;  (Task  III)  Questionnaire 
Development  and  Dissemination;  (Task  IV)  Analysis  of  Question¬ 
naire  and  Follow-up  Action;  (Task  V)  Analysis  and  Evaluation 
of  Integrated  Research  Results  Leading  to  Comprehensive  Recom¬ 
mendations  and;  (Task  VI)  Final  Report. 

1 . 4  STUDY  METHODOLOGY 

SAI  designed  a  highly  structured  methodology  for  this  study 
of  SQA  tools,  techniques  and  methods.  By  insisting  on  a  struc¬ 
tured  and  a  vigorous  approach  in  the  initial  stages  of  the 
project,  individual  project  tasks  were  created  which  integrate 
efficiently  to  yield  the  desired  result.  This  subsection  is 
organized  according  to  the  six  tasks. 

1.4.1  Task  I  -  Project  Kick-Off  and  Background  Research 
Investigation/AnalysTs 

There  were  three  major  objectives  of  this  first 
task:  (1)  Conduct  meetings  to  introduce  project  personnel 

to  the  Review  Panel  composed  of  members  from  RADC,  DLA, 

ESD,  and  AFCMD  and  finalize  project  work  plan  and  schedule; 
(2)  Review  Government  furnished  materials  including  regu¬ 
lations,  manuals  and  standards  to  determine  the  appropriate 
boundaries  for  the  remainder  of  the  study;  and  (3)  Inves¬ 
tigate  software  acquisition  and  contract/management  guidance 
documentation,  related  specifically  to  software  quality. 

This  investigation  represented  the  major  level  of  effort 
in  this  task.  It  was  one  of  three  project  research  efforts 
designed  to  gain  a  clear  understanding  of  current  software 
quality  tools,  techniques  and  communication  methods  before 
proceeding  to  the  integrated  analysis.  This  investigation 
of  documents  also  served  as  a  baseline  for  the  preparation 
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of  the  other  two  research  tools  of  this  project,  the 
Structured  Data  Collection  Instrument  and  the  Structured 
Questionnaire.  Throughout  the  contract  updated  materials 
were  supplied  by  the  Review  Panel.  A  preliminary  analysis 
of  the  guidance  documents  was  submitted  to  the  Contracting 
Officer's  Technical  Representative  (COTR). 

1.4.2  Task  II  -  Structured  Data  Collection  Instrument 

Development  and  On-Site  Interview? 

There  were  two  major  objectives  of  this  second  task: 
(1)  Develop  Forms  for  a  Structured  Data  Collection  Instru¬ 
ment  and  (2)  Conduct  on-site  reviews  and  interviews  using 
this  Structured  Data  Collection  Instrument.  The  SAI  survey 
team  conducted  interviews  with  government  personnel  in 
accordance  with  the  schedule  arranged  by  RADC.  At  the  com¬ 
pletion  of  this  task  a  preliminary  analysis  of  the  reviews 
and  interviews  was  submitted  to  the  COTR. 


1.4.3  Task  III  -  Questionnaire  Development  and  Dissemin¬ 
ation 

There  were  two  major  objectives  of  this  third  task: 
(1)  Develop  a  questionnaire  with  instructions  and  (2) 
Disseminate  the  questionnaire  to  the  organizations  identi¬ 
fied  by  the  Government.  This  questionnaire  was  developed 
based  on  the  background  research  of  Task  I  and  preliminary 
analysis  of  the  on-site  reviews  and  interviews  of  Task  II. 
This  questionnaire  was  the  third  of  three  research  tools 
used  by  SAI  in  this  project.  SAI  met  with  COTR  to  gain 
approval  of  quest ionna i re  and  dissemination  strategy. 

The  questionnaire  was  disseminated  to  DLA,  ESD 
and  AFCMD  organizations  identified  by  the  Government  during 
the  questionnaire  approval  meetings.  The  completed  ques¬ 
tionnaires  were  returned  to  SAI  for  compilation  and  anal¬ 
ysis. 

1.4.4  T ask  IV  -  Analysis  of  Results  of  Questionnaire 
and  Fol  lcw-up~Act  i  on 

There  were  three  major  objectives  of  this  fourth 
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task:  (1)  Compile  the  results  of  the  returned  question¬ 

naires;  (2)  Follow-up  action  on  incomplete  or  unreturned 
questionnaires;  (3)  Analyze  results  of  all  completed 
questionnaires.  SAI  compiled  the  results  of  the  question¬ 
naire  in  workbooks  that  lead  to  customized  analysis  for 
each  organization. 

At  this  point,  all  of  the  research  was  completed 
and  a  preliminary  analysis  was  completed  in  three  areas: 

(1)  Investigation  of  Documents,  (2)  Structured  Data  Col¬ 
lection  Interviews  and  (3)  Questionnaire  Results.  SAI 
prepared  and  submitted  an  Interim  Technical  Report  con¬ 
cerning  these  areas. 

1.4.5  Task  V  -  Analysis  and  Evaluation  of  Integrated 

ive 

There  were  four  major  objectives  to  this  fifth  task: 
(1)  Integrate  the  results  of  all  prior  research;  (2) 

Analyze  and  evaluate  integrated  results;  (3)  Identify 
areas  for  improvement;  and  (4)  Make  specific  recommen¬ 
dations  . 

The  first  step  in  this  task  was  the  integration  of 
the  results  obtained  from  the  three  research  tools;  (1) 
Investigation  of  Documents,  (2)  Structured  Data  Collection 
interviews  and  (3)  Questionnaire.  This  integration  provided 
the  starting  point  for  a  total  analysis. 

SAI  analyzed  the  integrated  results  for  three  groups: 
DLA,  ESD,  and  AFCMD.  SAI  structured  the  analyses  using  a 
"Software  Quality  Analysis  Form".  The  form  structured  in¬ 
formation  such  as:  (l'J  Description  of  tool,  technique  or 
communication  method,  (2)  Current  Status,  (3)  Requested 
Changes,  (4)  Reason  behind  request  for  change.  This  in¬ 
tegrated  analysis  is  presented  in  Section  III  of  this 
report . 

SAI  evaluated  the  integrated  research  results  for 
the  three  groups.  SAI ' s  evaluation  was  performed  by  first, 
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Research  Results  Leading  to  Comprehens 
Recommendations 


developing  evaluation  criteria  and  second,  by  designing 
a  "Software  Quality  Evaluation  Form”  that  reflected  the 
evaluation  criteria. 

This  evaluation  was  structured  to  correspond  with 
the  analysis  form.  The  "Software  Quality  Evaluation  Form" 
structured  information  such  as:  (1)  Current  opinion, 

(2)  Advantages,  (3)  Disadvantages,  (4)  Advantages  of  change 
and  (5)  Disadvantages  of  change.  This  evaluation  is  pre¬ 
sented  in  Section  IV  of  this  report. 

SAI  identified  "Areas  for  Improvement".  SAI  struc¬ 
tured  a  "Software  Quality  Improvement  Form".  This  Form 
corresponded  with  the  evaluation  form.  The  "Software 
Quality  Improvement  Form"  structured  information  such  as: 
(1)  High  payback  changes,  (2)  Areas  for  Improvement  and 

(3)  Potential  Requirements.  These  "Areas  for  Improvement" 
are  presented  in  Section  V  of  this  report. 

SAI's  recommendations  were  developed  using  a  "Soft¬ 
ware  Quality  Recommendation  Form".  This  Form  structured 
the  information  such  as:  (1)  Effort  required  to  meet 
Potential  Requirements,  (2)  Benefit  and,  (3)  SAI's  Recom¬ 
mendations.  These  recommendations  are  presented  in  Section 
VI  of  this  report. 

1.4.6  Task  VI  -  Final  Report 

There  were  two  major  objectives  to  this  sixth  and 
final  task:  (1)  Completion  and  submittal  of  the  Final 
Report  and  (2)  Submittal  of  a  training  course  outline. 

The  training  course  ..as  submitted  under  separate  cover  to 
reflect  the  recommendations.  This  document  is  the  Final 
Report  and  is  structured  to  reflect  the  methodology  of 
Task  V.  The  analysis,  evaluation,  areas  for  improvement 
and  recommendations  are  presented  in  narrative  addressing 
all  the  points  brought  out  by  the  four  "Form"  methodology 
of  Task  V. 
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1.5  OTHER  CONSIDERATIONS 


Throughout  the  report,  those  persons  engaged  in  the  work 
of  software  quality  assurance  are  referred  to  as  "software 
quality  assurance  personnel",  a  term  consistent  with  that  used 
throughout  the  industry. 

"Management"  personnel  refers  to  those  staff  members  who 
determine  policies,  manage  software  quality  assurance  programs 
or  supervise  other  staff  members. 

"Operations"  personnel  refers  to  those  staff  members  who 
monitor  contractor  software  quality  assurance. 

In  addition,  the  subject  of  the  report  is  Government  soft¬ 
ware  quality  assurance  practices,  procedures  and  personnel.  In 
instances  where  discussion  refers  to  contractors,  "contractor' 
has  been  specifically  stated. 


SECTION  II 


ORGANIZATION  PROFILES  OF  STUDY  PARTICIPANTS 

2.1  PARTICIPATING  ORGANIZATIONS 

Representatives  of  the  following  organizations  participated 
in  this  study  during  both  the  on-site  interview  and  question¬ 
naire  data  collection  efforts: 

•  Defense  Logistics  Agency  (DLA) 

•  Defense  Contract  Administration  Services  Management 
Areas  (DCASMAs) 

•  Defense  Contract  Administration  Services  Plant 
Representative  Offices  (DCASPROs) 

•  Electronic  Systems  Division  (ESD/TO) 

•  ESD  System  Program  Offices  (SPOs) 

•  Air  Force  Contract  Management  Division  (AFCMD) 

•  Air  Force  Plant  Representative  Offices  (AFPROs) 

Figures  1  through  3  illustrate  the  organizational  struc¬ 
tures  (as  of  November,  1980)  of  the  Defense  Logistics  Agency, 
the  Air  Force  Contract  Management  Division  and  the  Electronic 
Systems  Division  with  their  respective  subordinate  organ¬ 
izations  . 

Since  the  time  of  the  data  collection  effort,  the  Electronic 
Systems  Division's  Technical  Operations  section  has  reorganized, 
specifically  in  the  software  and  hardware  directorates.  Changes 
are  not  reflected  on  a  new  organization  chart  but  the  area  of 
reorganization  can  be  determined  by  a  comparison  of  Figure  3 
with  Figure  4.  The  directorate  of  Engineering  and  Testing  (old 
TOE)  and  the  Directorate  of  Computer  Systems  Engineering  (old 
TOI)  have  combined  talents  into  the  new  TOE,  which  includes  the 
following  four  subgroups:.  (1)  TOEA  which  closely  approximates 
the  old  TOIQ,  (2)  TOEE  which  closely  approximates  the  old  TOlT, 
(3)  TOFT,  predominantly  a  hardware  division,  and  (4)  TOEO,  a 
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resource  management  office. 

TOEA  is  responsible  for  software  quality  assurance,  training 
and  computer  resource  estimation  (i.e.,  sizing  and  timing). 

TOEE  controls  programs  such  as  software  metrics,  and  com¬ 
puter  security,  and  supports  the  program  for  Ada  development. 

TOET  provides  experts  in  all  areas  of  hardware  functions, 
including  document  reviews. 

TOEO  is  the  support  office,  controlling  human  resource 
management . 

The  reorganization  tends  to  place  less  emphasis  on  software 
quality  assurance.  However,  the  efforts  of  software  quality 
assurance  specialists  have  increased  to  compensate  for  this 
decrease  in  emphasis. 

2.2  SELECTION  OF  PARTICIPANTS 

Organizations  specified  in  the  Request  for  Proposal  for  on¬ 
site  visits  and  personnel  interviews  were: 

•  Headquarters/Air  Force  Contract  Management  Division 

•  Two  Air  Force  Plant  Representative  Offices,  (1)  AFCMD 
Detachment  38,  General  Electric  Company,  RESD5SD, 

Valley  Forge,  PA,  (2)  AFCMD  Detachment  45  (EN)  ,  West- 
inghouse  Electric  Corporation,  Baltimore,  MD 

•  Headquarters/Defense  Logistics  Agency/QES 

•  DCASPRC ,  GTF./Sylvania,  Needham,  MA 

•  ESD/TOIQ  and  one  SPO,  COBRA/JUDY 

Individual  participants  interviewed  from  these  organizations 
were  selected  by  representatives  from  the  Defense  Logistics 
Agency,  the  Electronic  Systems  Division  and  the  Air  Force  Con¬ 
tract  Management  Division. 

Questionnaire  respondents  were  selected  from  the  head¬ 
quarters  of  the  Defense  Logistics  Agency,  the  Electronic  Systems 
Division,  the  Air  Force  Contract  Management  Division  and  from 
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their  respective  subordinate  organizations.  Personnel  from  the 
management  and  operations  levels  at  each  location  were  selected. 

2 . 3  HUMAN  RESOURCES  PROFILES 

A  tracking  of  the  number  of  personnal  authorized  and 
assigned  for  software  quality  assurance  over  a  five-year  period 
yielded  the  following  profile: 

•  Within  the  Defense  Logistics  Agency  and  the  Air  Force 
Contract  Management  Division  there  has  been  a  minimal 
increase  in  the  number  of  personnel  authorized  and 
assigned  to  software  quality  assurance. 

•  Within  the  Electronic  Systems  Division  there  has  been 
little  or  no  increase  in  the  number  of  personnel 
authorized  and  assigned  to  software  quality  assurance. 

Overall,  the  increase  in  the  number  of  personnel  assigned  to 
software  quality  assurance  indicates  that  more  emphasis  is  being 
placed  on  software  quality  assurance.  However,  manpower  is  still 
inadequate.  Respondents  stated  that  this  situation  may  be  alle¬ 
viated  by  hiring  qualified  software  quality  assurance  specialists 
and/or  by  providing  in-depth  training  to  those  personnel  who  are 
currently  performing  software  quality  assurance  activities  and 
have  the  added  advantage  of  experience  in  working  on  Government 
contracts . 

In  many  cases,  the  number  of  personnel  authorized  for  soft¬ 
ware  quality  assurance  is  not  available.  A  key  factor  affecting 
the  staffing  of  software  quality  assurance  projects  and  maintain¬ 
ing  staff  is  turnover.  The  major  causes  of  turnover  within  the 
Defense  Logistics  Agency,  the  Electronic  Systems  Division  and 
the  Air  Force  Contract  Management  Division  are  losses  to  private 
industry,  moves  within  the  agency,  military  transfers  and  re¬ 
tirement.  Recruitment  of  appropriately  skilled  personnel  is 
difficult  because  the  Government  is  competing  with  the  private 
sector  for  similar  skills. 
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SECTION  III 


ANALYSIS  OF  CURRENT  SOFTWARE  QUALITY 
ASSURANCE  PRACTICES  AND  REQUESTED  CHANGES 

3.1  OVERVIEW 

Information  concerning  current  software  quality  assurance 
practices  within  the  Defense  Logistics  Agency,  the  Electronic 
Systems  Division  and  the  Air  Force  Contract  Management  Division 
was  compiled  from  three  sources:  fl)  Responses  from  personnel 
interviewed  during  on-site  visits,  (2)  Information  from  question¬ 
naire  respondents,  and  (3)  Research  of  applicable  standards, 
regulations  and  specifications.  In  addition,  representatives 
from  these  three  major  organizations  contributed  valuable  in¬ 
formation  based  on  their  own  experiences  and  positions  within 
their  respective  agencies. 

The  analysis  of  all  information  indicates  that  software 
quality  assurance  practices  and  associated  problems  relate  to 
four  major  areas:  (1)  Government  Softuare  Quality  Assurance 
Guidance  Documents,  (2)  the  Relation  of  Software  Quality  As¬ 
surance  to  Software  Development,  (3)  Communication  Methods 
and  Model  Documents,  and  (4)  Training  and  Staffing. 

The  following  subsections  describe  by  area  the  current 
practices  and  changes  requested  by  members  of  each  organization. 
Where  information  is  essentially  the  same  for  the  three  organ¬ 
izations,  the  analysis  is  consolidated.  A  summary  of  requested 
changes  is  included  at  the  end  of  this  section.  This  forms  the 
basis  for  Section  IV,  Evaluation  of  Current  Software  Quality 
Assurance  Practices  and  Change  Requests. 

3.2  CURRENT  PRACTICES  RELATING  TO  GOVERNMENT  SOFTWARE 

Q'UAI.  iTV'TiSSTfRANuF  CU  I  DANCE  DOCUMEN  l  S 

The  discussion  of  software  quality  assurance  guidance  docu¬ 
ments  encompasses  five  areas:  (1)  Request  for  Proposal  Prepa¬ 
ration,  (2)  Contract  Award  Negotiations,  (3)  Memoranda  of  Agree¬ 
ment/Letters  of  Delegation,  (4)  Engineering  Change  proposals. 
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;iml  t  5 )  Subcontractor  (Control.  These*  arc  arcus  where  software 
quality  assurance  requir*  .icnts,  provisions  and  procedures  for 
implementation  should  he  addressed  in  detail  if  quality  soft¬ 
ware  is  expected. 

In  the  Defense  logistics  Agency,  tne  primary  guidance  docu¬ 
ment  for  developing  a  quality  assurance  management  plan  is  DLAM 
8200.1,  Procurement  Quality  Assurance  Manual.  Section  IX,  Part 
IS  "Procurement  Quality  Assurance  lor  Computer  Software"  provides 
direction  for  developing  a  software  quality  assurance  management 
plan. 

The  overall  guidance  documents  of  the  F.lectronic  Systems 
Division  is  APR  800-14,  Volume  I  -  Management  of  Computer  Re¬ 
sources  in  Systems,  and  Volume  II  -  Acquisition  and  Support  Pro¬ 
cedures  for  Computer  Resources  in  Systems.  Most  respondents 
stated  that  there  is  no  guidance  document  solely  for  developing 
software  quality  assurance  management  plans,  and  expressed  the 
opinion  that  one  is  needed.  At  the  System  Program  Office  level, 
the  Contractor’s  Computer  Program  Development  Plan  fCPDP)  is 
the  primary  management  plan. 

The  Air  force  Contract  Management  Division's  guidance  docu¬ 
ment  for  developing  a  quality  assurance  management  plan  is  AfCMDR 
“4-1,  Procurement  Quality  Assurance  Program.  Chapter  15  provides 
direction  for  developing  a  software  quality  assurance  management 
plan.  There  is  a  new  version  of  AFCMDR  74-1  dated  30  March 
1081,  but  this  version  was  not  mentioned  by  respondents.  Other 
guidance  documents  used  include: 

•  AFCMDR  178-1,  Contractor  Management  System  Evaluation 
Program  ('CMSFP'; 

•  AFR  800-14,  Volume  I  -  Management  of  Computer  Resources 
in  Systems,  and  Volume  II  -  Acquisition  and  Support 
Procedures  for  Computer  Resources  in  Systems 

In  addition  AFCMDR  800-1,  Contract  Management  engineering 
and  AICMPP  800-2  Cont  ract _ Management  engineering  (ini  do,  both 
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dated  August  1981  have  great  impact  on  AFCMD's  software  quality 
assurance  activities.  However,  these  two  documents  were  not 
mentioned  by  respondents  as  they  were  not  available  during  the 
data  collection  effort. 

Most  respondents  from  the  three  organizations  stated  that 
the  guidance  documents  are  adequate  for  making  management  deci¬ 
sions  and  for  assuring  the  quality  of  software  but  that  improve¬ 
ments  are  needed.  All  respondents  stated  that  software  quality 
assurance  should  be  more  heavily  emphasized. 

3.2.1  Request  for  Proposal  Preparation 

Software  quality  assurance  personnel  within  the 
Defense  Logistics  Agency  are  not  involved  in  request  for 
proposal  preparation. 

There  is  some  involvement  in  request  for  proposal 
preparation  within  the  Electronic  Systems  Division.  Criteria 
used  are: 

•  Request  for  proposal  review  guides  (for  example, 
ESD/TOI  01800-1) ; 

•  User  requirements; 

•  Military  standards; 

•  Available  funding;  and 

•  East  experience. 

Those  who  do  participate  in  request  for  proposal  reviews 
listed  the  following  as  valuable  contributions  to  be  made 
by  software  quality  assurance  personnel:  ensuring  that 
standards  and  regulations  are  followed;  ensuring  good 
engineering  practices,  and  checking  schedule  requirements. 

There  is  some  involvement  in  request  for  proposal 
preparation  within  the  Air  Force  Contract  Management  Divi¬ 
sion.  Participation  includes  planning,  writing  and  review. 
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Criteria  used  in  request  for  proposal  reviews  are: 


•  AFCMDR  74-1,  Procurement  Quality  Assurance 
Program ; 

•  MIL-S-52799A,  Software  Quality  Assurance 
Requirements ; 

•  MIL-Q-9858,  Quality  Program  Requirements; 

•  ASD/ESD  Software  Acquisition  Management 
Guidebooks ; 

•  Requests  of  PCOs ;  and 

•  Past  experience. 

Problems  occur  when  suggestions  are  made  by  software 
quality  assurance  personnel  and  no  response  is  received 
from  the  development  team.  The  Air  Force  Contract  Manage¬ 
ment  Division  is  currently  working  on  a  "feedback  loop" 
procedure  to  enhance  the  communication  system.  AFSCR  800-42, 
Program  Office/AFCMD  Interface  was  issued  on  1  May  1981 
which  "prescribes  the  policy  for  establishing  program  office/ 
AFCMD  interface  during  the  conceptual  phase  and  for  pro¬ 
viding  AFCMD  specialized  support  for  AFSC  program  offices 
throughout  the  acquisition  cycle". 

3.2.2  Contract  Award  Negotiation 

In  the  Defense  Logistics  Agency  and  the  Air  Force 
Contract  Management  Division  there  is  no  involvement  in 
contract  award  negotiations.  In  the  Electronic  Systems 
Division  those  few  who  do  participate  in  negotiations  listed 
as  procedures/guidelines  used:  matching  proposals  to 
technical  evaluation  criteria  and  tracing  proposals  back 
to  corresponding  request  for  proposals. 

3.2.3  Memorandum  of  Agreement/Letter  of  Delegation 

The  division  of  functions  between  the  Program  Office 
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and  the  Contract  Administration  Office  is  achieved,  gener¬ 
ally,  through  the  Memorandum  of  Agreement,  Letter  of  Dele¬ 
gation  and  Defense  Acquisition  Regulations  (for  example, 

DAR  20-703.3).  In  the  Defense  Logistics  Agency  the  division 
of  functions  is  not  always  negotiated.  Negotiations  that 
do  occur  range  from  the  time  of  review  of  requests  for 
proposals  to  six  months  after  contract  award.  Some  offices 
have  a  Letter  of  Delegation  with  a  Program  Office;  some  do 
not.  The  methods  of  deciding  the  division  of  functions 
include:  (1)  Delegation  of  responsibilities  by  the  Program 

Office;  (2)  Contract  clause  and/or  Quality  Assurance  Letters 
of  Instructions;  and  (3)  Directions  of  the  Program  Office 
through  the  ACO.  In  some  cases,  provisions  on  software  are 
specified;  in  others,  they  are  not  specified.  When  modifi¬ 
cations  of  the  provisions  on  soft’ware  occur  it  is  usually 
due  to: 

•  Attempts  to  meet  delivery  dates, 

•  Redefinition  of  DCASMA  functions. 

Within  the  Electronic  Systems  Division  there  was  no 
consistency  in  responses  regarding  the  existence  of  Memor¬ 
anda  of  Agreement,  Letters  of  Delegation  nor  on  the  pro¬ 
vision  of  software  instructions  contained  in  either.  Those 
who  were  aware  of  negotiations  stated  that  they  occur  at 
times  ranging  from  the  request  for  proposal  reviews  to  the 
selection  of  the  contractor. 

Within  the  Air  Force  Contract  Management  Division  the 
division  of  functions  is  usually  negotiated  at  times 
ranging  from  the  review  of  proposals  to  three  months  after 
contract  award.  Regulations/guidelines  followed  are: 

•  AFCMDR  800-11,  AFCMD  Memorandum  of  Agreement 
Management  System 
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•  AFSCR  74-1,  Quality  Assurance  Program 

•  AFCMDR  74-1,  Procurement  Quality  Assurance 
Program 

•  AFCMDR  178-1,  Contractor  Management  System 
Evaluation  Program 

•  AFSCP  800-3  A  Guide  for  Program  Management 
(Chapter  22) 

•  AFR  800-14,  Management  of  Computer  Resources 
in  Systems 

Some  offices  have  a  MOA  with  a  Program  Office;  others  do 
not.  In  some  cases,  provisions  on  software  are  specified; 
in  others  they  are  not  specified.  Occasions  when  modifi¬ 
cations  of  provisions  on  software  occur  include: 

•  Directions  from  the  Program  Office 

•  Changes  in  personnel  and  technical  expertise 

•  Updating  as  phases  of  programs  change 
5.2.4  Engineering  Change  Proposals  (ECPs) 

Generally  the  respondents  do  not  approve  Engineering 
Change  Proposals.  This  is  a  function  of  the  Program  Office 
which  may  be  delegated.  Within  the  Defense  Logistics 
Agency  the  DCAS-ACO  at  the  prime  may  approve  ECPs  when 
authority  is  delegated.  The  Engineering  and  Program  Support 
within  the  Air  Force  Contract  Management  Division  may  also 
approve  ECPs.  The  effects  noted  by  software  quality 
assurance  personnel  arc  increased  costs  and  delays  in 
implementation  due  to  the  time  lapse  between  the  submittal 
of  Engineering  Change  Proposals  and  approval.  Air  Force 
Contract  Management  Division  respondents  stated  that  approval 
for  class  II  ECPs  is  faster  since  they  usually  are  delegated 
this  responsibility  and  do  not  have  to  involve  the  Program 
Office  as  with  the  more  complicated  Ci’ss  I  ECPs. 
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Subcontractor  Control 


Most  respondents  agreed  that  subcontractor  control 
is  maintained  by  the  prime  contractor.  Several  respondents 
from  the  Defense  Logistics  Agency  and  the  Electronic  Systems 
Division  expressed  the  opinion  that  visibility  of  govern¬ 
ment  control  over  subcontractors  is  not  adequate.  Air  Force 
Contract  Management  Division  respondents  expessed  much 
dissatisfaction  with  subcontract  management.  Contractors 
should  maintain  control  via  reviews,  monitoring  and  reports; 
this  is  not.  always  done. 

3.2.6  Requested  Changes  Concerning  Software  Quality 
Assurance  Guidance  Documents 

Table  1  illustrates  all  specific  changes  requested  by 
respondents  for  the  improvement  of  government  software 
quality  assurance  guidance  documents  and  related  areas.  The 
organizations  from  which  a  change  request  originated  are 
indicated  in  the  appropriate  cell. 


TABLE  1.  REQUESTED  CHANGES  CONCERNING  GOVERNMENT 
SOFTWARE  QUALITY  ASSURANCE  GUIDANCE  DOCUMENTS 


' — Organizations  Requesting 

Changes 

DLA 

ESD 

AFCMD 

Requested  Changes  '  — . _ 

Place  greater  emphasis  on  government  soft¬ 
ware  quality  assurance  programs 

X 

X 

X 

Develop  a  DOD-wide  guidance  document 

X 

Develop  new  guidance  documents 

X 

X 

Revise  existing  guidance  documents  by  expand¬ 
ing  the  software  quality  assurance 
sections 

X 

X 

1 

l 

I 

x  i 

1 

Revise  plans  as  requirements  change 

X 

X 

i 

f 

_ L 

Provide  more  tools 

X 

i 

i 

Include  software  quality  assurance  personnel 
in  the  early  stages  of  acquisition 

X 

X 

> 

x  ! 
' 

Hire  evaluators  with  software  oriented 

skills 

X 

X 

X 

!  Provide  for  clear  definitions  of  responsi- 

|  bility 

X 

|  Improve  AFCMDR  800-1! 

X 

- J 
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3.3  RELATION  OF  SOFTWARE  QUALITY  ASSURANCE  TO  SOFTWARE 
DEVELOPMENT 

In  the  discussion  of  the  relationship  of  software  quality 
assurance  to  software  development,  the  following  areas  are  inte¬ 
grated  : 

•  Life  Cycle  Models 

•  Configuration  Management 

•  Baselining 

•  Reviews  and  Audits 

•  Documentation 

•  Tools  and  Techniques 

Although  treated  as  isolated  entities  during  the  interview 
sessions  and  in  the  questionnaire,  they  are  areas  that  should 
not  be  considered  alone  in  software  development  planning,  in 
life  cycle  model  construction  or  in  software  quality  assurance 
management  planning. 

3.3.1  Life  Cycle  Models 

Based  on  interviews  with  management  personnel  at 
Hq./DLA, the  life  cycle  model  is  used  only  for  illustration 
in  training  courses.  Operations  personnel  from  the  field 
stated  that  use  of  the  life  cycle  model  occurs  during  pro¬ 
curement,  after  contract  award  and  in  all  documentation 
from  the  contractors.  Questionnaire  respondents  from  both 
the  management  and  operations  levels  stated  that  they  work 
against  a  life  cycle  model.  The  most  frequently  cited 
models  were  the  Defense  Logistics  Agency's  and  the  contrac¬ 
tors  '  . 

Within  the  Electronic  Systems  Division  most  respon¬ 
dents  use  the  AFSC  life  cycle  model  from  AFR  800-14  and 
AFSCP  800-3.  The  life  cycle  model  is  predominantly  an 
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instruction  aid;  but,  according  to  the  interviews  con¬ 
ducted  with  management  personnel,  it  is  only  loosely 
followed  in  practice 

Most  respondents  from  the  management  and  operations 
level  within  the  Air  Force  Contract  Management  Division 
either  stated  that  they  do  not  work  against  a  life  cycle 
model  or  did  not  respond  at  all.  Those  respondents  that 
do  use  a  life  cycle  model  cited  the  AFSC  life  cycle  model 
from  AFR  800-14  and  AFSCP  800-3  and  the  Contractors'  as 
the  most  frequently  used. 

3.3.2  Configuration  Management 

Within  the  Defense  Logistics  Agency,  configuration 
management  is  generally  the  responsibility  of  the  contractor. 
Within  the  F.lectronic  Systems  Division  and  the  Air  Force 
Contract  Management  Division,  configuration  management 
responsibilities  are  shared  by  the  government  and  the 
contractors.  The  effectiveness  of  configuration  management 
plans  as  they  relate  to  software  acquisition  management/ 
software  quality  assurance  was  generally  rated  as  adequate. 
The  need  for  some  improvement  was  expressed. 

The  configuration  management  guideline  in  use  within 
the  electronic  Systems  Division  is  MIL-STD-483.  No  guide¬ 
lines  were  specified  by  Defense  Logistics  Agency  or  Air 
Force  Contract  Management  Division  respondents. 

3.3. 3  Basel Lnes 

According  to  interviewees  within  the  Defense  Logis¬ 
tics  Agency,  baselining  is  not  used  as  a  tool.  However, 
ques t i onna i re  respondent s  from  the  management  and  operations 
levels  stated  that  they  do  work  with  baselines.  Only 
the  operations  level  respondents  specified  Functional, 
Allocated  and  Product  Baselines. 


All  respondents  within  the  Electronic  Systems  Divi¬ 
sion  and  some  respondents  within  the  Air  Force  Contract 
Management  Division  stated  that  they  do  use  Functional, 
Allocated  and  Product  baselines  as  reference  points  in 
their  work  of  software  quality  assurance.  A  majority  of 
Air  Force  Contract  Management  Division  respondents  either 
stated  that  they  do  not  work  against  baselines  or  did  not 
respond  to  the  question. 

3.3.4  Reviews  and  Audits 

System  audits  and  reviews  are  witnessed  by  some 
respondents  with  the  aid  of  guidebooks  and  the  results  of 
previous  reviews.  Involvement  is  in  the  form  of  observation 
rather  than  an  active  participation.  Audits  most  frequently 
witnessed  are:  Physical  Configuration  Audits  and  Functional 
Configuration  Audits.  Reviews  most  frequently  witnessed 
are  System  Design  Reviews,  Preliminary  Design  Reviews,  Criti¬ 
cal  Design  Reviews,  and  Formal  Qualification  Reviews.  Other 
reviews  were  cited  but  none  are  used  on  a  widespread  basis. 

3.3.5  Documentation 

Contractor  documentation  is  reviewed  by  operations 
level  software  quality  assurance  personnel.  A  variety  of 
regulations  and  specifications  are  referenced  but  few 
methods  of  recording  results  were  cited  by  the  Defense 
Logistics  Agency,  the  Electronic  Systems  Division,  or 
the  Air  Fo.rce  Contract  Management  Division. 

3.3.6  Tools  and  Techniques 

With  the  exception  of  life  cycle  models  and  base¬ 
lines,  software  quality  assurance  personnel  use  very  few 
tools  independent  of  the  contractor. 

Checklists,  both  standard  and  improvised,  guidelines 
and  regulations  are  some  of  the  primary  independent  tools 
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used  to  evaluate  procedures.  These  tools  are  generally 
used  to  evaluate  contractor  documentation. 

The  Air  Force  Contract  Management  Division  uses 
AFCMDR  178-1,  Contractor  Management  System  Evaluation  Pro¬ 
gram  (CMSEP)  as  a  tool  to  evaluate  the  contractor's  per¬ 
formance.  The  techniques  of  this  evaluation  are  highly 
structured.  Respondents  stated  that  a  separate  section 
on  software  quality  assurance  should  be  incorporated.  Some 
respondents  stated  that  CMSEP  is  too  detailed  and  time 
consuming  and  is  not  needed  by  skilled  specialists.  Other 
respondents  stated  that  the  detailed  procedures  of  CMSEP 
are  useful  for  less  skilled  personnel. 

The  contractor's  use  of  other  software  development 
tools  and  techniques  is  monitored;  but  none  on  a  widespread 
basis.  These  include:  simulators,  requirements  analysis 
tools,  library  control  tools,  software  metrics,  debuggers, 
and  walkthroughs. 

3.3.7  Requested  Changes  Concerning  the  Relation  of  Software 

Quality  Assurance  to  Software  Development 

Table  2  illustrates  all  specific  changes  requested 
by  respondents  concerning  the  relationship  of  software  qual¬ 
ity  assurance  to  software  development.  The  organization 
requesting  a  specific  change  is  indicated  in  the  appropriate 
cell. 
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~  -  ■ - Oi  gan  izat  ions  Requesting 

'  — _______Changes 

Requested  Changes  — — 

DLA 

ESD 

AFCMD 

Revise  life  cycle  model  to  picture 
accurately  the  life  cycle  of 
software  development* 

X 

Use  the  life  cycle  in  planning 

X 

Integrate  software  quality  assurance 

activities  with  the  life  cycle  model 

X 

X 

Provide  for  government  enforcement  of  con¬ 
tractor  configuration  management  plans 

X 

Involve  experienced  personnel  in  baselining 
activities 

X 

X 

Train  software  quality  assurance  personnel 
in  software  development 

X 

X 

X 

Provide  more  reviews 

.  x _ 

X 

X 

Address  software  in  Configuration 

Management  Plans 

| 

X 

Require  internal  reviews  by  contractors 

X 

*At  the  time  of  the  interviews  (November,  1980)  at  Hq.  DLA 
it  was  stated  that  the  life  cycle  model  was  up  for  revision 
in  six  months. 


3.4  COMMUNICATION  METHODS  AND  MODEL  DOCUMENTS 

Communication  methods  vary  widely  within  and  between  the 
Defense  Logistics  Agency,  the  Electronic  Systems  Division  and 
the  Air  Force  Contract  Management  Division.  Although  several 
methods  are  employed,  no  formal  structure  is  evident.  In  addition 
to  Memoranda  of  Agreement  and  Letters  of  Delegation,  some  of  the 
communication  methods  currently  in  use  include  telephoning, 
briefings,  board  meetings  and  panel  discussions. 

In  responding  to  questions  concerning  government  software 
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quality  assurance  guidance  documents  and  the  relation  of  software 
quality  assurance  to  software  development,  respondents  either 
implied  that  communication  methods  can  be  improved  or  offered 
specific  suggestions  for  improvement.  Informal  communication 
methods  (for  example,  telephoning  for  advice  or  assistance)  are 
effective.  However,  a  formalized  structure  of  communication 
flow  is  needed  in  order  to  make  all  project  staff  aware  of  major 
responsibilities  and  functions  involved,  who  is  responsible  and 
what  is  expected. 

Table  3  lists  all  specific  changes  requested  by  respondents 
to  enhance  communication  methods  and  provide  for  effective  com¬ 
munication  flow.  The  organizations  from  which  requests  originated 
are  indicated  in  the  appropriate  cell. 


Structure  the  communication  system 
Require  maintenance  of  r e c o r d s 

Emphasize  coordination  between  buying 
activities  and  software  quality 


X 


X 


X 


X 


_ assurance _ 

iistablish  points  of  contact _ X 

Provide  definite  guidelines  and  regulations _ 

Ii s t a blish  frequency  of  communication _ _ X 

Provide  for  more  frequent  communication _ 

Identify  experts  who  can  be  contacted  for 
technical  assistance 


X 


_X 

_X 

_X 

X 


X 


X 


3.5  TRAIN  INC.  AND  STAFFING 


Discussion  on  training  and  staffing  covers  four  areas:  (1) 
formal  education  of  respondents;  (2)  Training  courses  attended; 
(3)  Orientation  provided,  (4)  Certification  programs  available. 
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3.5.1  Formal  Education 

Formal  education  is  defined  as  high  school,  college 
or  university  courses  completed.  These  may  culminate  in  or 
lead  to  a  diploma  or  a  degree  or  be  non-degree  related.  The 
average  formal  education  of  respondents  is  described  by 
agency . 

3. 5.1.1  Defense  Logistics  Agency 

All  respondents  from  the  management  level 
have  had  from  two  to  four  years  of  college  education 
with  major  areas  of  concentration  in  business  or 
management.  Less  than  half  of  the  respondents  from 
the  operat ions  level  have  had  from  two  to  five 
years  of  college  education  with  major  areas  of  con¬ 
centration  in  science  or  engineering.  Although 
some  respondents  have  had  individual  computer 
science  courses,  none  have  been  trained  specifically 
in  this  area. 

3.5.1. 2  Electronic  Systems  Division 

All  respondents  from  the  program  management 
level  have  had  four  to  six  years  of  college  education 
with  major  areas  of  concentration  in  computer  science, 
electronics,  business  administration  or  math.  All 
respondents  from  the  operations  level  have  had  from 
two  to  seven  years  of  college  with  major  areas  of 
concentration  in  math  or  computer  systems  engineer¬ 
ing.  Only  one  respondent  had  taken  no  formal  courses 
in  computer  science. 

3 . 5 . 1 . 3  Air  Force  Contract  Management  Division 

All  respondents  from  the  management  level 
have  had  from  one  to  six  years  of  college  education 
with  major  areas  of  concentration  in  some  aspect  of 
engineering,  math,  or  science.  The  majority  of 
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respondents  from  the  operations  level  have  had  from 
one  to  seven  years  of  college  education.  Major 
areas  of  concentration  include:  engineering,  math, 
law,  management,  quality  control,  computer  science, 
electronics,  physics,  and  accounting.  Although 
some  respondents  have  had  some  computer  science/ 
software  courses,  a  greater  number  have  not. 

3.5.2  On-The-Job  Training 

On-the-job  training  includes  courses,  seminars, 
instructions,  etc.,  received  by  personnel  to  assist  them 
to  develop  the  skills  needed  to  perform  their  task  assign¬ 
ments.  On-the-job  training  received  by  respondents  is 
discussed  by  agency. 

3. 5. 2.1  Defense  Logistics  Agency 

Some  training  has  been  received  by  respon¬ 
dents  to  assist  them  in  performing  software  quality 
assurance  functions.  This  las  been  attained  through 
public  seminars,  courses  offered  within  the  Depart¬ 
ment  of  Defense,  DCAS  S-36  Procurement  Quality 
Assurance  for  Computer  Software,  and  courses  respon¬ 
dents  have  taken  on  their  own  initiative.  DLAM 
8220.5,  Quality  Assurance  Intern  Program,  has  been 
developed  by  DLA  but  does  not  include  software  quality 
assurance . 

3 . 5 . 2 . 2  Hlectronic  Systems  Division 

Training  in  software  quality  assurance 
functions  has  primarily  centered  on  reading  some/all 
of  the  PSD  Software  Acquisition  Management  Guidebooks 
and  attendance  at  HSD/TOI's  Computer  Technology  Trans¬ 
fer  Training  Center  course  on  Software  Acquisition 
Management.  Other  training  has  been  received  through 
seminars,  ART  courses  and  courses  offered  within  the 
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Department  of  Defense. 

3 . 5 . 2 . 3  Air  Force  Contract  Management  Division 

Some  training  has  been  received  by  respon¬ 
dents  to  assist  them  in  performing  software  quality 
assurance  functions.  Training  has  been  acquired 
through  DCAS  S-36  Procurement  Quality  Assurance  for 
Computer  Software,  AFCMD  Fundamentals  of  Software 
Quality  Assurance  SQA-1,  public  seminars,  and  courses 
within  the  Department  of  Defense. 

3.5.3  Orientation 

Orientation  includes  any  entry  level  instruction 
received  by  respondents  to  assist  them  in  their  software 
quality  assurance  functions.  Orientation  is  discussed  by 
agency. 

3 . 5 . 3 . 1  Defense  Logistics  Agency 

No  respondents  from  the  management  level 
and  less  than  half  from  the  operations  level  received 
orientation  instruction  pertaining  to  their  software 
quality  assurance  functions.  Among  the  orientation 
instructions  listed  were: 

•  SQA  course,  DCAS  S-36,  Procurement 
Quality  Assurance  for  Computer  Soft¬ 
ware  , 

•  Section  IX,  Part  15,  of  DSAM  8200.1, 
Procurement  Quality  Assurance  Manual, 

•  Introduction  to  other  quality  assurance 
personnel  on  staff, 

•  Staff  assistance  in  developing 
Procedure  Checklists. 
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3. 5. 3. 2  Electronic  Systems  Division 

Some  respondents  received  orientation  per¬ 
taining  to  software  acquisition  management/software 
quality  assurance;  others  did  not.  Among  the  orien¬ 
tation  instructions  listed  were: 

•  ESD/TOI  Software  Acquisition  Manage¬ 
ment  Course, 

•  USN  SQA,  Computer  Software  for  Tech¬ 
nical  Personnel, 

•  Development,  Control  and  Acquisition. 

3. 5. 3. 3  Air  Force  Contract  Management  Division 

Some  respondents  received  orientation  per¬ 
taining  to  software  acquisition  management/ soft  - 
ware  quality  assurance;  the  majority  did  not.  Among 
the  orientation  instructions  listed  were: 

•  DCAS  S-36,  Procurement  Quality  Assur¬ 
ance  for  Computer  Software, 

•  AFCMD  Fundamentals  of  Software  Quality 
Assurance,  SQA-1 

•  Internal  APPRO  orientation, 

•  USN  SQA,  Computer  Software  for  Tech¬ 

nical  Personnel 

•  AFCMDR  74-1,  Procurement  Quality 
Assurance  Program 

•  Introduction  to  Software. 

3.5.4  Certification  Programs 

Certification  programs  according  to  DLAM  8220.4, 
Quality  Assurance  Certification  Program  arc  programs  that 
lead  to  formal  recognition  that  an  individual  is  technically 
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qualified  in  a  specific  area.  Hlectronic  Systems  Division 
respondents  stated  there  are  no  certification  programs  for 
software  quality  assurance  specialists.  Three  respondents 
from  the  Air  Force  Contract  Management  Division  cited 
51XX  AFSC  as  a  specialty  code  describing  qualifications 
for  officers.  Correspondence  courses  through  Extension 
Course  Institute  provide  some  training  in  computer  science. 

Within  the  Defense  Logistics  Agency  conflicting 
views  were  expressed.  The  majority  of  management  respon¬ 
dents  stated  that  there  are  no  certification  programs 
for  software  quality  assurance  management  specialists. 

Those  who  stated  there  are  described  the  course  of  action 
toward  certification  as  formal  training,  and  experience  in 
accordance  with  DLAM  8220.4,  Quality  Assurance  Cert i f icat ion 
Program. 

Approximately  half  of  the  operations  respondents 
stated  that  there  are  certification  programs  for  software 
specialists.  The  course  of  action  toward  certification 
was  described  as: 

•  Courses  listed  in  DLAH  8220.1,  Quality  Assur¬ 
ance  Technical  Development  Program,  Course 
Catalog , 

•  Outside  training  at  private  institutions, 

•  S-38,  Computer  Fundamentals, 

•  S-36,  Procurement  Quality  Assurance  of 
Computer  Software, 

•  Completion  of  required  or  equivalent  courses 

and  one  year  of  hands-on  experience  in  the  field. 


3.5. 5  Requested  Changes  Concerning  Training  and  Staffing 

Table  4  illustrates  all  specific  changes  requested 
by  respondents  concerning  training  and  staffing.  The 
organizations  requesting  a  specific  change  are  indicated 
in  the  appropriate  cell. 
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TABLE  4.  REQUESTED  CHANGES  CONCERNING  TRAINING  AND  STAFFING 


TRAINING 

Provide  structured  training  to  software 

quality  assurance  personnel  including: 

•  management  oriented  courses 

•  software  engineering 

•  software  quality  assurance 
Make  training  available  to  more  personnel 
ORIENTATION 

Conduct  seminars/workshops 

Include  specifications  applicable  to  software 

Instruct  on  software  quality  assurance  system 
control  techniques 

Familiarize  personnel  with  contractor 

organization,  methodology  and  facilities 

Work  with  experienced  software  managers  on  a 
daily  basis 

Explain  the  role  of  the  Plant  Representative 
in  software  acquisition  management 

Instruct  on  software  quality  assurance  tools 
and  techniques 

Explain  the  requirements  of  MI  L-S-52779/ 

Explain  software  quality  assurance  funda¬ 
mentals  and  management  of  a 
software  acquisition  program 

CERTIFICATION  PROGRAMS 

Require  that  certified  personnel  be 
assigned  to  projects 

Develop  structured  certification  programs 


X 

7 


X 


3 . 6  SUMMARY  OF  REQUESTED  CHANGES 

The  following  subsections  consolidate  all  requested 
changes  submitted  by  respondents  from  the  Defense  Logistics 
Agency,  the  Klectronic  Systems  Division  e^d  the  Air  Force 
Contract  Management  Division.  Changes  are  grouped  under  four 
major  areas:  (1)  Government  Software  Quality  Assurance  Guidance 
Documents,  (2)  Relation  of  Software  Quality  Assurance  to  Soft¬ 
ware  Development,  (3j  Communication  Methods  and  Model  Documents, 
and  (4)  Training  and  Staffing. 

Generally,  all  changes  requested  by  respondents  are  listed. 

A  particular  change  may  have  been  requested  by  one  person  or 
several  respondents.  In  some  cases,  requested  changes  may  seem 
contradictory,  but  they  were  submitted  from  various  organizations 
with  different  practices  and  procedures. 

3.6.1  Requested  Changes  For  Improvement  of  Software 

Quality  fls~ surance ~7Tu  IcTa nee  Documents 

Requested  changes  from  the  Defense  Logistics  Agency, 
electronic  Systems  Division  and  the  Air  Force  Contract 
Management  Division  concerning  software  quality  assurance 
guidance  documents  and  related  areas  are  summarized  as 
fol lows : 

•  Place  greater  emphasis  on  government  software 
quality  assurance  programs 

•  Hire  software  quality  assurance  evaluators 
with  software  oriented  skills 

•  Provide  for  clear  definitions  of  respons ibil ity 

•  Improve  Memoranda  of  Agreement/Letters  of 
Delcgat  ion 

•  Revise  existing  documents  by  expanding  the 
software  quality  assurance  sections 

•  Revise  plans  as  requirements  change 
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Include  software  quality  assurance  personnel  in 
the  early  stages  of  the  acquisition  process 


•  Develop  new  guidance  documents 

•  Develop  a  DoD-wide  management  plan 

3.6.2  Requested  Changes  For  Improvements  in  Relation  of 
Software  Quality  Assurance  to  Software  Development 

Requested  changes  from  the  Defense  Logistics  Agency, 
Electronic  Systems  Division  and  the  Air  Force  Contract 
Management  Division  concerning  the  relation  of  software 
quality  assurance  to  software  development  are  as  follows: 

•  Use  life  cycle  model  in  planning 

•  Integrate  software  quality  assurance  activities 
with  the  life  cycle  model 

•  Address  software  in  configuration  management 
plans 

•  Provide  for  government  enforcement  of  contrac¬ 
tor  configuration  management  plans 

•  Involve  experienced  personnel  in  baselining 

activities 

•  Provide  more  reviews 

3.6.3  Requested  Changes  For  Improvement  of  Communication 
Methods  and  Model  Document's 

Requested  changes  from  the  Defense  Logistics  Agency, 
Electronic  Systems  Division  and  the  Air  Force  Contract 
Management  Division  concerning  communication  methods  are 
summarized  as  follows: 

•  Structure  the  communication  system 

•  Provide  definite  regulations  and  guidelines 
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•  Emphasize  coordination  between  buying  activities 
and  software  quality  assurance  activities  by: 

(1)  establishing  points  of  contact  and  (2) 

I 

establishing  frequency  of  communication 

•  Require  that  records  be  maintained. 

3.6.4  Requested  Changes  For  Improvement  of  Training 
and  Staffing 

Requested  changes  from  the  Defense  Logistics  Agency, 
Electronic  Systems  Division  and  the  Air  Force  Contract 
Management  Division  concerning  training  and  staffing  are 
summarized  as  follows: 

3.6.4. 1  Training 

•  Make  training  avialable  to  more 
personnel 

•  Develop  structured  training  courses 
in : 

(1)  Management  of  software  projects 

(2)  Software  Engineering 

(3)  Software  Quality  Assurance 

3. 6. 4. 2  Certification  Programs 

•  Develop  structured  accredited  certi¬ 
fication  programs 

•  Require  that  certified  personnel  be 
assigned  to  projects 

3.6.4. 3  Orientation 

•  Familiarize  personnel  with  contractor 
organization,  methodology  and  facili¬ 
ties 

•  Establish  communication  system  and 
points  of  contact 
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•  Work  with  experienced  government  soft¬ 
ware  managers  on  a  daily  basis 

•  Instruct  on  software  specifications 
applicable  to  the  particular  project 

•  Instruct  on  software  quality  assurance 
system  control  techniques  tailored 

to  the  contract 

•  Establish  the  particular  tools  and 
techniques  applicable  to  the  project. 

Respondents  were  cooperative,  brief  and 
frank.  Many  suggestions  and  requests  for  changes 
were  submitted,  which  were  pertinent  to  a  particular 
organization,  division  or  project.  However,  through¬ 
out  the  analysis,  issues  were  raised  crossing  all 
areas  and  organizations  and  these  should  be  ad¬ 
dressed.  In  Section  IV,  evaluation  of  the  major  areas 
for  which  changes  were  requested  includes:  identi¬ 
fication  of  problems,  examination  of  possible  so¬ 
lutions,  and  suggestions  for  implementation. 


SECTION  IV 


EVALUATION  OH  CURRENT  SOFTWARE  QUALITY  ASSURANCE 
PRACTICES  AND  CHANGE  REQUESTS 


4 . 1  OVERVIEW 

From  the  integrated  analysis  of  current  software  quality 
assurance  practices  and  the  compilation  of  changes  requested,  four 
major  areas  of  concern  are  found  to  exist  throughout  the  Defense 
Logistics  Agency,  the  Electronic  Systems  Division  and  the  Air 
Force  Contract  Management  Division.  These  areas  are: 

(1)  Government  Software  Quality  Assurance  Guidance 
Documents , 

(2)  Relation  of  Software  Quality  Assurance  to  Software 
Development , 

(3)  Communication  Methods  and  Model  Documents, 

(4)  Training  and  Staffing. 

Concerns,  generally,  are  not  limited  to  any  one  agency  or  organi¬ 
zation  but  pervade  all  three.  In  the  following  subsections, 
criteria  for  each  area  are  presented.  In  light  of  these  criteria, 
the  advantages  of  each  requested  change  are  determined. 

Implementation  of  some  of  the  requested  changes  will  create 
problems  that  should  be  considered.  The  problems  are  not  unique 
to  any  one  change  request,  but  relate  to  the  entire  concept 
of  improving  the  practices  and  procedures  associated  with  the 
software  quality  assurance  areas  listed  above.  A  summary  of 
associated  problems  is  presented  at  the  end  of  each  area  of 
discussion. 

4 . 2  PURPOSE  OF  SOFTWARE  QUALITY  ASSURANCE:  GUIDANCE  DOCUMENTS 

A  guidance  document  serves  as  a  guide  for  decision  making  in 
developing  a  software  quality  assurance  management  plan.  Not  all 
elements  of  a  model  plan  are  necessary  for  all  projects;  in  fact, 
it  is  one  of  the  tasks  of  management  to  select  which  aspects  of 
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the  activity  will  be  formally  planned  and  controlled  and  which 
will  be  informal  or  ad  hoc.  The  selection  criteria  are  simply  the 
necessity  versus  the  cost.  In  general,  formality  is  most  costly 
in  terms  of  time  and  money;  therefore,  if  It  is  not  necessary,  it 
should  not  be  exercised.  On  the  other  hand,  lack  of  formal  plan¬ 
ning  and/or  controls  carries  the  risk  of  chaos.  These  can  easily 
lead  to  problems  that  are  far  more  expensive  than  the  controls 
that  would  have  prevented  them.  In  general,  a  wel 1 -written ,  clear, 
concise  software  quality  assurance  management  plan  will: 

•  Provide  for  clear  division  of  responsibilities  during 
early  system  acquisition  phases, 

•  Identify  appropriate  standards/guidelines  to  be  followed, 

•  Determine  appropriate  computer  software  quality 
assurance  requirements, 

•  Develop  software  quality  assurance  evaluation  procedures, 

•  Provide  for  corrective  action  procedures,  and 

•  Establish  an  efficient  communication  system. 

In  light  of  these  criteria,  the  following  subsections  discuss 
the  advantages  of  and  problems  associated  with  the  changes  re¬ 
quested  in  regard  to  software  quality  assurance  guidance  documents. 
The  purpose  is  not  to  address  each  of  these  criteria  at  this  time. 
Each  change  request  is  examined  in  order  to  determine  areas  where 
criteria  are  not  being  met  and  where  improvements  can  be  made. 

4.2.1  Place  Greater  Emphasis  on  Government  Software  Quality 
Assurance  Programs 

Many  software  projects  have  been  unsuccessful  because 
of  an  inadequate  software  quality  assurance  program.  A 
greater  emphasis  on  the  software  quality  assurance  discipline 
is  an  important  aspect  for  ensuring  that  computer  program  de¬ 
sign,  code,  associated  documentation,  and  performance  comply 
with  contractual  requirements.  These  aims  are  accomplished 
by  implementing  a  system  of  controls  that  are  applied  to  all 
phases  and  aspects  of  the  software  development  process.  This 
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assumes  that  specific  software  quality  requirements  are 
clearly  specified  in  requests  for  proposals  and  resultant 
contracts . 

The  benefits  derived  from  the  application  of  software 
quality  assurance  disciplines  are  directly  related  to  how 
well  the  software  quality  assurance  disciplines  are  defined 
and  put  into  effect.  The  value  of  those  disciplines  is  re¬ 
flected  by  the  overall  efficiency  of  the  development  program 
and  the  ultimate  quality  of  the  software. 

Procedures  developed  for  hardware  quality  assurance  do 
not  satisfy  the  objectives  of  software  quality  assurance. 

Few  similarities  exist  between  evaluating  the  quality  of 
tangible  equipment  and  evaluating  software  products  such  as 
documentation  where  adherence  to  documentation  standards,  and 
documentation  review  are  the  key  elements. 

The  benefits  of  placing  deserved  emphasis  on  software 
quality  assurance  are  obvious.  Requirements  will  be  speci¬ 
fied  in  contracts.  Documentation  of  program  design,  coding 
and  testing  will  be  structured  according  to  requirements. 
Hrrors  will  be  discovered  and  corrective  action  will  be 
applied  at  the  earliest  appropriate  stage.  Cost  overruns  and 
schedule  delays  will  be  reduced  and  an  improved  quality  soft¬ 
ware  will  be  the  end  product. 

4 . 1 . 1  Provide  for  Clear  Definitions  of  Responsibility 

A  most  essential  factor  in  the  development  of  manage¬ 
ment  plans  is  matching  tasks  to  personnel  resources.  A  pro¬ 
ject  cannot  develop  successfully  if  the  technical  expertise 
for  carrying  out  responsibilities  and  functions  is  not  avail¬ 
able.  Managers  must  be  aware  of  contractual  requirements  in 
order  to  assess  that  each  task  falls  within  the  expertise  of 
his  or  her  staff.  When  Memoranda  of  Agreement  and  Letters  of 
Delegation  are  required,  communication  between  System  Program 
Offices  (SPOs)  and  Contract  Administration  Offices  (CAOs) 
should  be  initiated  early  in  the  acquisition  cycle 
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and  continue  throughout  the  life  of  the  contract  in  order 
that  personnel  needs  are  accurately  assessed  and  met. 


There  are  several  advantages  derived  from  defining 
responsibilities.  The  possibility  of  omitting  tasks  is 
limited.  Personnel  will  perform  tasks  that  are  within  their 
level  of  expertise.  Needed  specialized  skills  will  be 
determined  and  delegated  or  acquired  before  the  project 
begins.  Each  staff  member  will  be  fully  aware  of  his  or 
her  own  particular  functions  and  those  of  other  staff  mem¬ 
bers.  The  result  will  be  an  efficiently  run  project  that 
adheres  closely  to  schedule  and  cost  constraints. 

4.2.3  Improve  Memoranda  of  Agreement  (MOAs)  and  Letters  of 

Delegation  (LODs) 

Memoranda  of  Agreement  and  Letters  of  Delegation  that 
contain  specific  provisions  on  software  will  enforce  the 
accomplishment  of  contractual  requirements.  Technical  repre¬ 
sentatives  will  be  assigned  according  to  their  expertise  and 
their  specific  tasks  will  be  outlined  in  detail.  MOAs  and 
LODs  will  be  written  based  on  communication  between  the  SPOs 
and  CAOs.  Tasks  will  not  be  delegated  to  CAOs  that  do  not 
have  the  expertise  available  to  perform  them.  This  should 
help  SPOs  and  CAOs  in  identifying  areas  of  needed  training 
for  their  personnel. 

4.2.4  Revise  Existing  Guidance  Documents 

Software  quality  assurance  guidance  documents  currently 
in  use  throughout  the  Defense  Logistics  Agency,  the  Electronic 
Systems  Division,  and  the  Air  Force  Contract  Management  Divi¬ 
sion  contain  some  basic  elements  but  are  heavily  dependent  on 
procedures  already  established  for  hardware.  Software  is  a 
much  more  intangible  commodity  to  assess.  Yet,  the  quality 
of  software  may  determine  the  success  or  failure  of  a  project. 
Stringent  controls  are  necessary  throughout  software  develop¬ 
ment.  These  controls  can  only  be  exercised  through  a  de¬ 
tailed,  structured  model  plan  devised  specifically  for 


software  quality  assurance.  A  structured  plan,  by  its  very 
nature,  will  be  detailed,  specifically,  to  software  and  will 
integrate  the  applicable  tools  and  techniques  at  the  appro¬ 
priate  phases  of  tne  life  cycle.  It  would  only  remain  to 
integrate  the  individual  procedures  and  controls  that  are 
specific  to  a  project. 

Expansion  of  existing  guidance  documents  will  build 
upon  past  work  and  integrate  with  plans  already  developed. 
Interruptions,  slowdowns  or  even  stoppage  of  work  caused  by 
the  introduction  of  new  procedures  and  acclimating  all  per¬ 
sonnel  will  be  minimal.  Adverse  cost  effects  will  be  less 
drastic.  In  addition,  personnel  will  not  be  subjected  to  the 
frustrations  and  often  the  resistance  involved  in  adjusting 
to  new  procedures  and  methods  when  all  that  is  desired  is 
more  detail. 

4.2.5  Revise  Plans  As  Requirements  Change 

Not  all  elements  of  a  model  software  quality  assurance 
management  plan  may  apply  to  every  software  development  pro¬ 
ject,  or,  depending  on  the  complexity  of  a  project,  they  may 
not  apply  to  the  same  degree.  In  addition,  software  quality 
assurance  is  receiving  greater  emphasis  in  the  software  in¬ 
dustry  which  is  leading  to  more  intensified  research  to 
develop  tools  for  enhancing  software  quality.  Resulting 
advances  in  technology  will  be  evident  by  specification  of 
these  advances  as  requirements. 

Software  quality  assurance  guidance  documents  must  be 
structured  for  the  purpose  of  control  and  yet  be  flexible 
enough  to  permit  adaptability  to  technological  advances  and 
the  resultant  changes  in  requirements.  In  other  words,  if  a 
guidance  document  serves  as  a  "guide"  in  developing  software 
quality  assurance  management  plans,  it  will  be  detailed 
specifically  to  software,  but  allow  for  selection  and  inte¬ 
gration  of  applicable  tools  and  techniques  without  a  major 
revision  of  the  plans.  If  this  can  be  accomplished, 
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delays  associated  with  revisions  of  regulations,  specifica¬ 
tions  and  standards  should  be  non-existent. 

4.2.6  Include  Software  Quality  Assurance  Personnel  In 

Early  System  Acquisition  Stages 

Project  success  depends  upon  the  amount  and  quality  of 
planning  that  occurs  before  the  Full  Scale  Engineering 
Development  Phase.  All  staff  who  are  to  be  involved  in  the 
project  should  be  represented  in  planning  sessions.  There  is 
apparently  little/no  involvement  of  software  quality  assurance 
personnel  in  RFP  preparation,  proposal  review,  and  contract 
award  negotiation.  If  responsibility  for  software  quality 
assurance  is  to  be  accepted  and  performed  by  professionals, 
then  the  same  level  of  professionals  should  be  involved  in 
establishing  initial  specifications,  reviewing  responses  to 
proposals  and  ensuring  that  requirements  are  included  in 
contracts.  Early  input  from  software  quality  assurance 
personnel  would  aid  in  eliminating  future  problems.  Based  on 
their  past  experiences  with  contractor  software  quality 
assurance  procedures,  government  software  quality  assurance 
personnel  can  make  recommendations  which,  if  incorporated  in 
RFP's,  will  aid  contractors  in  addressing  quality  assurance 
requirements  and  submitting  clearly  defined  proposals,  plans 
and  procedures.  Further,  at  the  time  of  contract  award  nego¬ 
tiations,  continued  involvement  of  software  quality  assurance 
personnel  will  ensure  that  software  quality  assurance  pro¬ 
cedures  are  not  overlooked  or  that  the  impact  of  any  de¬ 
emphases  which  are  considered  are  fully  understood. 

4.2.7  Develop  New  Model  Software  Quality  Assurance 

Guidance  Documents 

There  are  several  advantages  in  developing  new  guidance 
documents  specifically  for  software  quality  assurance. 
Development  of  a  new  model  plan  will  eliminate  the  problems 
which  have  arisen  by  following  hardware  oriented  procedures 
to  assure  quality  software.  Contractors  are  often  dependent 
on  government  contracts.  Therefore,  if  the  government 
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initiates  a  planning  effort  in  this  area,  contractors  will  be 
likely  to  follow  suit.  Rather  than  depend  on  contractors  to 
develop  software  quality  assurance  plans  to  meet  contractual 
requirements,  the  government  should  take  the  lead  in  requiring 
software  quality  controls  to  meet  the  advances  in  current 
technology  and  reduce  rising  software  costs. 

As  the  focus  on  software  quality  assurance  increases, 
emphasis  should  be  placed  on  formation  of  a  separate  group 
responsible  for  quality  related  concerns  and  detached  from 
the  developmental  group.  Personnel  skilled  in  software  areas 
will  be  needed  to  implement  the  new  software  quality  assurance 
plans.  They  can  be  made  available  through  recruitment,  or 
through  the  development  of  formal  structured  training  programs 
to  enhance  the  technical  expertise  of  those  software  quality 
assurance  personnel  who  have  been  trained  in  other  areas,  for 
example,  hardware. 

4.2.8  Develop  DoD-Wide  Model  Software  Quality  Assurance 

Management  Plan 

One  internal  tri-service  operating  procedure  for 
assuring  software  quality  provides  distinct  advantages.  A 
standardized  plan  will  enable  the  government  to  develop  a 
standardized  contract  strategy  with  regard  to  software  quality 
assurance.  Contractors  will  become  accustomed  to  standard 
government  operating  procedures  for  evaluating  the  quality  of 
software.  Areas  that  formerly  may  have  been  deemphasized  will 
receive  greater  attention.  For  example,  in  the  development  of 
a  standardized  contract  strategy,  subcontractor  control  pro¬ 
cedures  for  software  quality  assurance  will  be  specified  in 
greater  detail.  Control  on  the  part  of  the  government  and 
the  contractor  will  be  more  visible. 

Memoranda  of  Agreement  and  Letters  of  Delegation  will 
improve.  SPOs  from  the  three  services  will  develop  standard 
practices  in  communicating  with  all  CAOs .  The  resulting  MOAs/ 
LODs  will  be  written  with  greater  facility. 
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4.2.9  Problems  Associated  With  Requested  Changes  Concerning 

Government  Software  Quality  Assurance  Guidance 

Documents 

Placing  greater  emphasis  on  software  quality  assurance 
may  result  in  increased  costs  initially.  If  software  quality 
assurance  requirements  are  specified  by  contract,  contract 
funding  must  reflect  this.  Adjustments  may  have  to  be  made 
in  project  schedules.  Additional  requirements  regarding 
software  quality  assurance  and  evaluation  procedures  may 
necessitate  lengthening  the  time  alloted  for  project  comple¬ 
tion.  However,  through  specifying  software  quality  assurance 
requirements  by  contract,  reviews  and  evaluations  will  dis- 
,  cover  errors  at  the  earliest  appropriate  stage.  This  will 

result  in  reducing  the  costs  that  later  error  detection  may 
entail.  Early  error  detection  and  corrective  action  will 
also  minimize  schedule  delays. 

Two  factors  must  be  determined: 

(1)  Will  the  additional  costs  incurred  by 
specifying  requirements  in  the  contr.  ct  be 
offset  by  the  cost  savings  that  will  result 
from  early  error  detection? 

(2)  Will  the  end  product  of  a  greater  quality  of 
software  justify  an  increase  in  costs? 

■  The  development  of  new  software  quality  assurance 

guidance  documents  within  each  organization  will  require  the 
combined  efforts  of  personnel  at  all  levels.  This,  in  turn, 
will  involve  the  absorption  of  time  that  ordinarily  would  be 
given  to  other  work.  Inadequate  manpower  is  already  a  common 

<  complaint.  To  release  personnel  from  their  software  quality 

assurance  tasks  may  be  detrimental  to  the  software  products. 
To  hire  new  personnel  as  replacements  may  not  be  feasible. 
i  To  assign  tasks  to  other  software  quality  assurance  special¬ 

ists  may  be  to  overload  already  overworked  staff.  This,  too, 

■  would  be  detrimental  to  the  software  products. 


46 


4.2.9  continued 


An  alternative  solution  is  to  hire  the  services  of 
outside  consultants  to  develop  new  software  quality  assurance 
guidance  documents.  The  primary  factor  is  cost.  The  cost  of 
hiring  outside  consultants  must  be  weighed  against  the  cost 
of  using  in-house  personnel  and  the  possible  effects  on  soft¬ 
ware  quality  assurance  work  that  must  continue. 

To  disregard  existing  plans  may  result  in  the  loss  of 
valuable  work  that  has  already  been  generated  and  only  re¬ 
quires  further  development.  The  resultant  product  may  involve 
a  duplication  of  time,  effort,  and  plans  that,  again,  already 
exist  and  only  need  expansion. 

Expansion  of  the  software  quality  assurance  sections 
of  existing  guidance  documents  will  involve  the  same  resources 
and  attendant  problems  as  those  stated  above.  Decisions  will 
have  to  be  made  regarding  the  use  of  in-house  personnel  or 
hiring  outside  consultants.  However,  the  total  effort  may  not 
be  as  great  and  the  total  cost  may  be  less  expensive.  It  is 
important  for  each  organization  to  consider  beforehand  the 
elements  that  are  lacking  in  their  current  guidance  documents 
as  determined  by  this  study,  and  the  amount  of  effort  that 
will  be  involved  in  a  revision.  It  is  possible  that  revision 
will  evolve  into  the  development  of  new  documents  and  the 
necessary  resources  for  this  work  should  be  pre-planned. 

The  development  of  a  DoD-wide  model  software  quality 
assurance  management  plan  will  require  intensive  planning 
efforts.  Representatives  from  all  organizations  concerned 
must  be  included.  The  groundwork  in  this  direction  was  under¬ 
taken  at  the  Joint  Logistics  Commanders  (JLC)  Software  Work¬ 
shop  held  in  April,  1979.  Recommendations  and  plans  for 
implementation  are  already  developed.  Costs  of  implementa¬ 
tion  may  be  considerable  both  in  development  of  uniform 
guidance  documents,  regulations,  standards  and  procedures, 
and  in  implementation  within  all  organizations.  Several 

factors  must  be  considered: 
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•  Will  the  costs  of  developing  and  implementing 
DoD-wide  plans  and  procedures  regarding  software 
quality  assurance  be  offset  by  eventual  reduced 
costs  in  software  development? 

•  Will  the  costs  involved  in  revising  existing 
documents  be  greater  than,  equal  to,  or  less  than 
developing  new  documents  within  each  organization? 

•  Can  the  determination  be  made  as  to  which  plans, 
DoD-wide  or  specific  organization  plans,  will 
effect  a  greater  quality  of  software? 

4.2.10  Summary 

Whether  or  not  existing  plans  are  expanded,  new  plans 
are  developed  or  a  uniform  DoD-wide  plan  is  adopted,  it  is 
essential  that  a  software  quality  assurance  guidance  document 
become  a  unique  document.  As  such  it  will  contain: 

•  Software  quality  assurance  requirements  that 
should  be  required  by  contract, 

•  Provisions  for  staffing  projects  with  the 
technical  expertise  required  at  all  levels, 

•  References  to  standards/guidelines  specifically 
written  for  software, 

•  Evaluation/corrective  action  procedures 
specifically  developed  for  software  quality 
assurance,  and 

•  Methods  of  communication  among  all  government 
personnel  involved  in  software  quality  assurance 
for  the  project. 

Once  this  objective  has  been  attained,  then  the  work  of  im¬ 
plementing  software  quality  assurance  practices  throughout 
the  software  development  process  will  be  performed  with 
greater  facility. 
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4 . 3  RELATION  OF  SOFTWARE  QUALITY  ASSURANCE  TO  SOFTWARE  DEVELOPMENT 

Software  development  extends  from  the  identification  of  the 
system  requirements  to  product  "buy-off"  and  subsequent  in-field 
operation. 

During  the  software  development,  software  quality  assurance 
provides  assessments,  evaluations,  and  reviews  of  the  software 
products.  Software  quality  assurance  also  conducts  many  of  the 
actual  software  control  functions.  Some  of  the  controls  covered 
by  software  quality  assurance  consist  of  the  following: 

•  Incorporation  of  software  quality  assurance  into  the 
overall  software  program  planning, 

•  Preparation  and  evaluation  of  standards  that  guide  the 
preparation  of  software  documentation,  design,  code, 
validation  and  verification, 

•  Evaluation  of  the  software  design  process  and  design 
products  for  conformance  to  requirements, 

•  Monitoring  of  the  software  design  for  compliance  with 
design  and  performance  requirements,  adequacy  of 
methods  used,  and  positive  evidence  of  compliance, 

•  Review  of  software  test  requirements,  plans  and 
procedures  for  compatibility  and  adequacy, 

•  Monitoring  of  software  tests  for  conformance  with 
procedures,  and 

•  Implementation  of  a  system  for  recording,  reporting, 
and  tracking  software  problems  and  for  assuring  the 
adequacy  of  corrective  actions. 

In  light  of  the  above  criteria,  the  following  subsections 
discuss  the  advantages  of  and  problems  associated  with  the  changes 
requested  regarding  the  relation  of  software  quality  assurance  to 
software  development.  Again,  the  purpose  is  not  to  address  each 
of  these  criteria  at  this  time.  Each  change  request  is  examined 
in  order  to  determine  areas  where  criteria  are  not  being  met  and 
where  improvements  can  be  made. 

49 


4.3.1  Use  Life  Cycle  Model  In  Planning 

Authorities  in  the  software  field  agree  on  the  impor¬ 
tance  of  a  life  cycle  model  in  planning  and  implementing  a 
software  development  program.  Life  cycle  models  are  often 
graphic  representations  of  all  development  activities;  and, 
if  structured  correctly,  work  in  conjunction  with  the  manage¬ 
ment  plan.  A  typical  life  cycle  model  will  be  divided  into 
phases  of  software  development  and  each  phase  partitioned 
into  more  specific  functional  activities.  Input  data,  output 
data  including  documentation,  and  appropriate  quality  assur¬ 
ance  activities  will  be  specified.  Use  of  a  life  cycle  model 
will  ensure  that:  milestones  are  established  and  goals  are 
set  by  all  project  members;  no  activities  are  omitted;  proper 
documentation  is  prepared  on  time;  and  a  system  of  controls 
is  defined  to  accomplish  project  objectives. 

4.3.2  Integrate  Software  Quality  Assurance  Activities  In 

Life  Cycle  Models 

Graphic  depictions  of  where  software  quality  assurance 
activities  "fit"  into  the  software  development  process  pre¬ 
sent  a  total  picture  to  the  software  quality  assurance 
personnel  and  aid  them  in  preparing  and  channeling  efforts, 
tools  and  techniques  in  the  appropriate  direction.  Software 
quality  assurance  personnel  working  with  such  a  model  will 
review  and  evaluate  the  approach,  the  methods,  the  status, 
and  the  achievements  during  each  software  development  stage. 
Performance  of  these  activities  during  the  proper  stage  could 
generate  a  list  of  problem  areas  requiring  immediate  solution. 
Assessment  of  problems  and  application  of  corrective  action 
early  in  a  project  has  proved  to  minimize  the  negative  effects 
on  cost  and  schedule  performance  involved  with  later  error 
detection. 

4.3.3  Address  Software  In  Configuration  Management  Plans 

In  conjunction  with  life  cycle  development,  configura¬ 
tion  management  identifies,  baselines,  controls,  and  reports 
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changes  to  software  products.  Configuration  management  plans 
are  logical  extensic.is  of  overall  management  plans.  Software 
is  a  unique  commodity  and  should  be  included  as  such  in  con¬ 
figuration  management  plans.  If  this  is  accomplished  early 
in  the  acquisition  process,  needed  changes  will  be  discovered 
and  implemented.  Costly  errors  that  are  discovered  later  in 
software  development  will  be  minimal. 

4.3.4  Provide  for  Government  Enforcement  of  Contractor 

Configuration  Management  Plans 

Although  configuration  management  plans  may  vary  from 
contractor  to  contractor,  the  development  of  procedures  for 
maintaining  software  stability  and  controlling  change  should 
be  contractual  requirements;  and,  therefore,  subject  to  en¬ 
forcement.  If  software  quality  assurance  personnel  are  in¬ 
volved  in  Request  for  Proposal  preparation  and  Contract  Award 
negotiations,  they  could  ensure  that  the  configuration  of 
computer  programs  and  all  associated  documents  are  controlled 
by  the  identification  and  baselining  of  configurations. 

Change  control  and  accounting  procedures  to  assure  proper 
implementation  and  visibility  could  be  developed  for  software 
changes . 

4.3.5  Involve  Software  Quality  Assurance  Personnel  In 

Baselining  Activities 

In  the  development  of  software  life  cycle  plans  and 
models,  the  establishment  of  baselines  is  an  integral  part. 
Establishing  baselines  in  the  system  life  cycle  development 
is  done  by  the  contractor  subject  to  review  and  approval  by 
the  government;  however,  software  quality  assurance  personnel 
can  play  an  important  role  in  evaluating  baseline  documents. 
Baselines  are  references.  On  a  life  cycle  model,  each  rep¬ 
resents  a  point  where  the  software  system  definition  is 
formally  reviewed,  agreed  upon,  and  published  as  the  new 
reference . 
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System  baseline  documents  are  the  tangible  products  of 
the  system  development  process.  They  are  the  principal  com¬ 
munications  media  within  the  contractor's  organization  and 
between  the  contractor  and  the  acquisition  agency.  It  is  the 
purpose  of  the  software  quality  assurance  staff  to  ensure 
that  the  baseline  documents  provide  structure  for  attaining 
and  maintaining  the  visibility  necessary  to  control  change, 
properly  audit  the  system  to  ensure  integrity  and  provide  all 
concerned  with  detailed  information  concerning  the  status  of 
the  system  at  any  point  in  time. 

4.3.6  Provide  More  Reviews 

If  software  quality  assurance  activities  are  integrated 
with  life  cycle  models  and  software  development  planning, 
areas  where  reviews  are  required  will  be  evident.  Every 
phase  of  software  development  should  include  adequate  reviews. 
In  expanding  the  quality  assurance  sections  of  guidance 
documents  caution  should  be  exercised  in  developing  more 
reviews.  There  may  be  an  adequate  number  of  reviews,  but 
the  manner  in  which  they  are  conducted  may  be  inadequate. 

Formal  and  informal  reviews  are  required  and  are  con¬ 
ducted  in  contracted  software  development  projects.  However, 
software  quality  assurance  evaluators  generally  monitor  con¬ 
tractor  reviews.  If  software  quality  assurance  requirements 
are  specified  by  contract,  the  task  of  software  quality- 
assurance  personnel  is  to  ensure  that  requirements  are  ful¬ 
filled.  This  means  they  must  conduct  reviews  and  evaluations 
independent  of  contractor  reviews.  Independent  reviews  and 
evaluation  tools  developed  and  applied  by  software  quality- 
assurance  staff  may  uncover  errors  overlooked  by  contractors. 
As  has  been  emphasized,  discovery  of  errors  at  the  appropriate 
stage  and  application  of  corrective  action  will  save  the  time 
and  costs  that  later  error  detection  will  cause. 
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4.3.7  Problems  Associated  With  Requested  Changes  Regarding 

the  Relation  of  Software  Quality  Assurance  to  Software 

Development  "  ~ 

The  primary  difficulty  with  regard  to  improving  the 
relation  of  software  quality  assurance  to  software  develop¬ 
ment  is  the  planning  effort.  Resources  will  have  to  be  ded¬ 
icated  to  developing  a  model  life  cycle  with  software  quality 
assurance  activities  integrated. 

The  participants  in  the  development  of  model  reviews, 
audits  and  documentation  procedures  should  be  specifically 
selected,  based  on  the  goal  to  be  achieved  and  their  ability 
to  contribute  to  achieving  that  goal.  Again  decisions  will 
have  to  be  made  with  regard  to  the  use  of  in-house  personnel 
or  outside  consultants.  The  factors  to  be  considered  are 
essentially  the  same  as  those  for  consideration  in  making 
decisions  on  improving  software  quality  assurance  guidance 
documents;  namely,  the  scope  of  work  involved,  the  projected 
costs  of  the  effort  and  the  effects  of  both  on  ensuring  an 
improved  quality  of  software. 

4.3.8  Summary 

The  development  of  quality  software  will  most  likely 
occur  only  if  the  controls  necessary  for  assuring  quality 
are  planned  and  specified  as  requirements.  Specification  of 
requirements  will  enable  the  software  quality  assurance  group 
to  exert  a  more  effective  influence  in  the  development  of 
quality  software.  In  order  for  this  to  occur  it  is  necessary 
that : 

•  Software  is  addressed  in  contractor  configuration 
management  plans, 

•  Software  quality  assurance  personnel  are  in¬ 
volved  in  evaluation  of  baseline  documentation, 

•  Standards  are  developed  that  guide  the  prepara¬ 
tion  of  software  documentation,  design,  code, 
validation  and  verification, 
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•  Standard  checklists  are  developed  for  reviews, 
assessments  and  evaluations  of  software  products, 

•  Developed  standards  are  adaptable  to  specific 
contracts,  and 

•  A  system  for  recording,  reporting  and  tracking 
software  problems  is  implemented. 

Once  the  software  quality  assurance  guidance  document 
is  developed  and  life  cycle  development  with  integrated  soft-  > 
ware  quality  assurance  activities  is  structured,  then  communi¬ 
cation  methods,  including  specifications  for  documentation 
and  documentation  reviews,  and  requirements  for  maintaining 
records  must  be  clearly  defined. 

4.4  COMMUNICATION  METHODS  AND  MODEL  DOCUMENTS 

Most  of  the  comments  here  are  directed  toward  formal  communi¬ 
cations.  Informal  methods  of  communication  are  important  and  are 
effective,  but  are  more  loosely  constructed.  Many  of  the  short-  j 
comings  in  a  software  project  can  be  traced  in  part  to  misunder-  j 
standings  resulting  from  communication  failures.  By  maintaining  j 
a  clearly  defined  and  coherent  communication  system,  these  problems  j 
can  be  reduced  with  positive  effects  on  the  software  development 
project. 

There  are  two  essential  methods  to  reduce  communication 
failures.  The  first  is  to  reduce,  as  much  as  possible,  the  number 
of  communications  required.  This  is  accomplished  by  partitioning 
tasks  and  simplifying  interfaces  and  interchanges.  The  second  is 
to  provide  reliable,  readily  available  communciation  channels  and 
mechanisms  wherever  needed. 

In  a  software  development  project,  review  of  documentation 
is  the  primary  means  of  assessment  and  evaluation.  Software  offers 
no  visibility  except  through  its  associated  documentation.  The 
quality  of  documentation  is  directly  related  to  the  quality  of 
software.  Superior  documentation  increases  the  probability  of  1 

the  quality  of  delivered  software.  Poor  documentation  decreases  I 
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the  probability  of  quality  of  the  delivered  software.  Hence, 
documentation  is  the  primary  form  of  communication  between  the 
developer,  the  software  quality  assurance  staff  and  the  end  user. 

For  effective  communication  to  occur  it  is  essential  that: 

•  The  purpose  of  the  communication  be  stated, 

•  The  methods  of  communicat ion  be  structured, 

•  All  project  staff  be  aware  of  the  lines  of  communication 
and  their  particular  responsibilities,  and 

•  If  possible,  the  time  communication  is  required  is 
specified . 

In  light  of  these  criteria,  the  following  subsections  discuss 
the  advantages  of  and  problems  associated  with  the  changes  re¬ 
quested  for  improvement  of  communicaitons .  Each  change  request  is 
examined  in  order  to  determine  areas  where  criteria  are  not  being 
met  and  where  improvements  can  be  made. 

4.4.1  Structure  the  Communication  System 

A  communication  system  is  a  necessary  means  of  linking 
the  various  parts  of  a  large  organization.  By  formally  de¬ 
fining  methods  and  channels  of  communication,  confusion  is 
minimized.  Distortion  due  to  mishandling  of  information  is 
reduced.  A  good  communica iton  system  can  ensure  that  the 
important  information  is  transmitted  and  the  unimportant  is 
discarded.  If  the  person  at  the  end  of  the  communication 
channel  receives  only  information  which  he  knows  is  important, 
that  information  will  receive  proper  attention  and  not  stag¬ 
nate  in  a  bottomless  pile.  This  is  time  efficient  for  all 
involved . 

4.4.2  Provide  Definite  Regulations  and  Guidelines 

Ciuidel  ines  and  regulations  are  essential  to  ensure  a 
reliable  and  uniform  communication  system.  If  the  proper 
guidelines  are  established  and  followed,  lines  of  communica¬ 
tion  will  be  determined  and  transmissions  will  be  released 
on  time,  and  to  the  proper  people. 
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Documentation  requirements,  design  reviews  and  configu¬ 
ration  control  are  integrally  related  to  communication. 
Specific  formats  for  documents  should  be  available  prior  to 
initiation  of  the  attendant  development  phase  if  a  good 
product  is  expected.  Considerable  manpower  can  be  saved  if 
the  engineers  know  exactly  what  they  must  produce  in  the  way 
of  documentation  and  exactly  how  and  when  to  produce  it.  If 
documentation  is  planned,  associated  reviews  and  audits  should 
discover  errors  early  on  in  each  phase,  provide  for  correc¬ 
tion,  save  time  and  costs  and  allow  a  project  to  remain  on 
schedule.  Audits  should  be  conducted  as  official  audits  to 
evaluate  conformance  of  prepared  documentation  to  contractual 
requirements . 

4.4.3  Emphasize  Coordination  Among  All  Project  Staff 

The  importance  of  establishing  lines  of  communication 
among  staff  members  has  been  emphasized.  Each  participant 
must  be  aware  of  each  other  participant's  responsibilities 
and  who  is  to  be  contacted  when  necessary.  The  problem  of 
overlooking  a  staff  member,  though  unintentional,  could  occur 
and  a  standardized  communication  system  will  remedy  this. 

Staff  support  in  management-related  matters  is  essential. 

The  management  process,  like  all  others,  is  interactive.  The 
results  of  that  interaction  depend  on  the  awareness  and  coop¬ 
eration  of  all  parties  involved.  If  management  has  not 
clearly  defined  the  functions  to  be  performed,  this  important 
interaction  cannot  take  place. 

In  establishing  clearly  defined  channels,  experts  can 
be  identified  to  whom  others  can  turn  for  assistance  in 
solving  problems.  This  is  a  factor  relative  to  informal 
rather  than  formal  communication.  If  a  staff  member  has  a 
problem  and  knows  exactly  whom  to  contact,  time  would  be 
saved  and  fewer  people  would  be  involved.  Although  sound  in 
theory,  this  is  difficult  to  accomplish.  Identification  of 
experts  requires  thorough  familiarity  with  the  organization 
and  its  resources. 


4.4.4  Problems  Associated  With  Requested  Changes 

For  Improvement  of  Communication  Methods 

and  Model  Documents 

Project  managers  must  maintain  a  high  degree  of  aware¬ 
ness  of  the  status  of  all  project  matters.  No  matter  how 
complete  and  concise  a  project  plan  may  be,  there  is  a  need 
for  adjustments  and  explicit  decisions  every  day.  If  the 
communication  system  is  too  rigidly  structured,  some  things 
may  be  compromised;  for  example,  the  flexibility  needed  to 
meet  emergencies. 

There  is  a  problem  associated  with  too  many  regulations 
and  guidelines.  If  staff  people  are  flooded  with  directives, 
there  is  confusion  regarding  which  to  follow  and  the  tendency 
may  be  to  ignore  the  directives  and  improvise. 

Standardizing  the  communication  system  can  be  time- 
consuming.  In  planning  and  implementing  any  new  policies 
and  procedures,  unforeseen  problems  may  surface.  The  tendency 
may  be  to  dispense  with  the  new  and  revert  to  the  former  ways 
of  operating.  An  initial  period  of  adjustment  is  needed  be¬ 
fore  the  effects  of  a  smoothly  running  communication  system 
are  realized. 

4.4.5  Summary 

In  establishing  a  communication  system,  it  is  important 
to  provide  for  both  formal  and  informal  methods.  Formal  com¬ 
munication  emphasizes  the  dissemination  of  information. 
Usually,  these  are  written.  Informal  communication  refers  to 
interpersonal  communications  between  two  or  more  people;  for 
example,  face-to-face  meetings  or  telephone  conversations. 

Both  types  of  communication  are  important  and  both  can  be 
planned . 

In  establishing  a  formal  communication  system  it  is 
important  to  determine: 

•  The  purpose  of  the  communication; 

•  The  sender  and  the  recipient; 
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the  communication  media; 

•  Follow-up  action,  if  any; 

•  Records  to  be  maintained,  if  any;  and 

•  Timeliness  of  communications. 

If  these  determinations  are  made  before  a  project  be¬ 
gins,  policies  and  procedures  will  be  facilitated  for  handlin 
major  issues  and  problems.  The  informal  communications  that 
occur  on  a  routine  daily  basis  will  provide  for  adjustments 
in  minor  areas . 

What  must  be  emphasized  is  that  no  guidance  documents, 
life  cycle  development  plan  nor  communication  system,  no 
matter  how  well  planned,  will,  of  themselves,  ensure  the 
quality  of  software.  The  key  factor  for  development  and 
implementation  is  software  quality  assurance  personnel  who 
possess  the  required  expertise.  The  necessity  of  providing 
the  required  software  oriented  skills  to  the  software 
quality  assurance  staff  is  imperative. 

4.5  TRAINING  AND  STAFFING 

Throughout  the  interview  sessions  and  questionnaire  re-  j 

sponses,  there  was  overwhelming  agreement  among  respondents  on 
the  issue  of  training.  In  every  area  of  questioning,  respondents  J 
emphasized  this  issue.  Personnel  felt  greater  involvement  on 
the  part  of  software  quality  assurance  personnel,  both  on  the 
management  level  and  the  operations  level,  will  be  of  little 
benefit:  (1)  If  training  is  not  provided  to  those  already  in 

government  service  who  are  making  efforts  to  perform  their 
functions  adequately,  and  (2)  If  personnel  with  training  in  soft¬ 
ware  areas  are  not  hired.  It  is  evident  that  software  quality 
assurance  personnel  are  conscientious  regarding  their  work  and 
are  aware  that  education  is  essential  if  the  quality  of  software 
is  to  improve. 

The  need  for  training  cannot  be  over-emphasized.  If  soft¬ 
ware  quality  assurance  personnel  do  not  have  the  technical  exper- 
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tise  to  ensure  that  software  quality  assurance  requirements  are 
met  in  each  of  the  successive  baselining  evaluations,  the  resul¬ 
ting  effects  on  the  software  product  may  be  devastating. 

Despite  the  fact  that  planning  the  system  development  and  estab¬ 
lishing  baselines  are  not  functions  of  the  software  quality 
assurance  staff,  knowledge  of  these  software  areas  is  essential 
for  effective  evaluation  to  be  performed.  It  is  unreasonable 
to  expect  evaluators  to  assess  performance  if  their  technical 
expertise  is  not  on  the  level  of  their  counterparts  on  the  con¬ 
tractor's  staff.  Again,  survey  respondents  themselves  have 
addressed  this  issue  throughout  all  areas  as  one  needing  serious 
consideration . 

The  following  subsections  discuss  the  advantages  of  and 
problems  associated  with  the  changes  requested  for  improvement 
of  training  and  staffing.  bach  change  request  is  examined  in 
order  to  determine  where  improvements  can  be  made. 

4.5.1  Make  Training  Available  to  More  Personnel 

The  most  important  resource  in  developing  and  imple¬ 
menting  a  software  quality  assurance  plan  is  skilled  per¬ 
sonnel.  Managers  and  operations  personnel  with  software 
oriented  skills  have  the  capability  to  specify  requirements, 
independently  assess  a  contractor's  plans  for  fulfilling 
contractual  requirements  and  to  evaluate  whether  or  not  the 
plan  is  being  followed  and  implemented  successfully.  Through 
unforeseen  circumstances,  requirements  may  change,  but  by 
following  a  structured  and  concise  plan,  skilled  managers 
and  operations  personnel  will  be  able  to  make  adjustments 
quickly  and  reduce  loss  of  time  and  added  costs. 

Again,  software  is  an  intangible  commodity  with  its 
own  particular  problems  and  methods  of  solution.  Personnel 
familiar  with  these  areas  through  past  experience  and/or 
through  training  will  anticipate  occurrences,  where  possible 
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and  offer  technical  advice  when  unforeseen  situations  arise. 

Since  an  overwhelming  number  of  respondents  expressed 
the  need  for  training  in  the  area  of  software  engineering, 
it  must  be  assumed  that  managers  and  operations  personnel 
are  experiencing  difficulty  in  assuring  the  quality  of  soft¬ 
ware  because  they  lack  technical  expertise  or  they  are  not 
current  with  the  state-of-the-art.  Certainly,  those  involved 
on  a  daily  basis  with  software  quality  assurance  should  be 
able  to  interact  with  contractor  personnel  whom  they  are 
monitoring.  If  such  is  the  case,  the  level  of  the  quality 
of  software  should  increase. 

4.5.2  Develop  Structured  Training  Courses 

Software  development  in  industry  and  government  is 
growing  at  an  incredible  rate.  The  number  of  people  needed 
in  work  requiring  direct  participation  in  software  develop¬ 
ment  activities  is  increasing  and,  therefore,  the  oppor¬ 
tunities  for  trained  software  professionals  are  expanding. 
Enlightened  managers  now  recognize  the  importance  and  the 
risks  associated  with  software  development,  and  few  projects 
of  substantial  size  are  successfully  accomplished  without 
explicit  planning,  control,  and  a  high  degree  of  visibility 

Unfortunately,  software  projects  that  meet  schedule, 
cost,  and  performance  objectives  are  still  the  exception  and 
not  the  rule.  Some  of  the  most  common  complaints  are: 

•  Delivered  Software  is  far  behind  schedule,  costs 
more  than  budgeted,  and  performs  inadequately; 

•  Software  is  constructed  and  documented  so  poorly 
that  it  is  almost  impossible  to  maintain  and 
enhance  at  reasonable  costs;  and 

•  Software  is  unreliable  and  requires  constant 
maintenance  to  correct  errors  discovered  during 
operation . 
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The  explanation  of  the  industry's  poor  record  does 
not  lie  in  the  fundamental  difficulty  of  the  work.  The 
actual  difficulty  has  been  in  matching  the  appropriate 
resources  to  the  problem.  This,  in  turn,  derives  from  two 
factors : 

Inability  to  identify  and  appreciate  all  aspects 
of  the  problem;  and 

Poor  packaging  of  the  resources,  i.e.,  the  required 
talents  are  distributed  over  a  large  group  of  indi - 
viduals . 

Software  engineering  and  management  education  can 
serve  to  alleviate  the  first  of  these  factors  by  exposing 
personnel  to  the  full  scope  of  the  software  development 
environment,  and  the  second,  by  preparing  personnel  with  a 
better  cross  section  of  abilities  with  which  to  operate  in 
that  environment. 

4.5.3  Establish  Certification  Programs 

If  the  government  is  unable  to  compete  with  industry 
in  hiring  software  managers,  certification  programs  to  train 
software  specialists  can  be  an  alternative  solution.  Certi¬ 
fication  programs  have  several  advantages.  They  will  be 
geared  to  Government  systems  and  acquisition  procedures  and 
provide  standard  training.  They  will  provide  a  level  of 
depth  at  the  technical  level  that  will  give  the  software 
quality  assurance  monitors  the  expertise  and  confidence 
necessary  to  deal  with  contractor  personnel.  If  the  govern¬ 
ment  takes  a  lead  role  in  recognizing  that  computer  soft¬ 
ware  is  a  commodity  as  is  electronics,  aeronautics,  etc.  , 
and  establishes  certification  programs  toward  that  end, 
contractors'  confidence  in  government  software  quality 
assurance  performance  will  be  enhanced.  The  contractors'  own 
software  quality  assurance  plans  will  reflect  the  added 
pressure  to  provide  quality  software  within  schedule  and 
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cost  parameters.  Finally,  certification  programs  will 
encourage  and  facilitate  further  in-depth  training  and 
emphasis  in  this  area. 

4.5.4  Provide  Orientation  Instruction 

Orientation  instruction  is  necessary  regardless  of 
the  amount  of  training  or  technical  expertise  obtained  by 
personnel.  Orientation  is  specific  to  a  facility  and  the 
program.  Familiarization  with  contractors’  organizations, 
methodologies  and  lines  of  communication  will  enable  the 
software  quality  assurance  personnel  to  plan  procedures 
that  can  be  accomplished  efficiently.  Orientation  instruc¬ 
tion  will  provide  personnel  with  a  brief  description  of 
the  system,  the  specifications  applicable  to  the  system, 
and  the  work  already  accomplished.  For  personnel  new  to 
Government  acquisition  procedures,  instruction  in  the  role 
of  the  software  quality  assurance  personnel  in  software 
acquisition  management,  and  briefings  on  related  military 
documents  will  provide  the  parameters  within  which  soft¬ 
ware  skills  are  utilized  and  again  will  save  time  that  may 
be  wasted  in  learning  on  an  "as  needed"  basis. 

4.5.5  Problems  Associated  With  Requested  Changes 

Concerning  Training  and  Staffing 

Software  quality  assurance  personnel  are  sometimes 
assigned  to  projects  requiring  specialized  skills  they  do 
not  have.  During  the  course  of  the  project,  they  may  request 
and  may  be  provided  some  training  to  enable  them  to  perform 
their  tasks.  However,  the  project  continues  and  by  the 
time  training  is  completed,  it  is  too  late  to  be  applied. 

The  best  time  to  receive  training  is  not  on  an  "as  needed” 
basis,  even  though  this  is  far  more  preferable  than  not  to 
receive  any  training  at  all. 

Technological  advances  in  software  development  are 
resulting  in  improved  automated  tools  for  design,  coding 
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and  testing.  Thi~  is  creating  the  need  for  software  quality 
assurance  personnel  with  more  specialized  skills  for  review¬ 
ing  and  evaluating  products  generated.  Training  in  the 
basic  principles  of  software  engineering  supplemented  by 
specialized  software  quality  assurance  skills  should  be  in 
place  already.  In  developing  plans  for  training,  consider¬ 
ation  should  be  given  to  training  software  quality  assurance 
personnel  in  the  performance  of  specific  activities  during 
specific  life  cycle  phases.  Specialization  may  directly 
affect  the  quality  of  software. 

Acquiring  skilled  software  quality  assurance  personnel 
is  a  particular  problem  for  the  Government.  Due  to  present 
job  classifications,  the  Government  is  unable  to  compete 
with  the  private  sector  for  scarce  resources.  To  further 
complicate  the  matter,  personnel  trained  in  software  quality 
assurance  often  leave  government  service  for  private  industry. 
This  could  be  repeated  upon  hiring  and  training  new  person¬ 
nel  with  the  accompanying  loss  of  time  and  money.  However, 
the  organizational  profile  developed  during  this  study  has 
revealed  that  many  of  the  CAO  personnel  are  dedicated  career 
employees.  Training  to  retread  these  government  personnel 
who  are  committed  to  government  service  should  result  in 
less  attrition.  This  could  prove  to  be  an  effective  stop¬ 
gap  measure  while  a  longer  range  solution  is  being  planned. 

4.5.6  Summary 

In  order  for  software  quality  assurance  personnel  to 
meet  the  increasing  advances  of  software  technology  and 
influence  the  quality  of  software  some  provisions  on  training 
must  be  implemented.  Training  should  cover  the  entire  ac¬ 
quisition  from  preparation  of  Requests  for  Proposals  through 
the  software  development  life  cycle.  The  most  important 
attribute  implied  by  the  term  "software  quality  assurance 
personnel"  is  that  of  a  problem  discoverer.  Through  cduca- 
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tion  in  software  management  and  software  engineering  orien¬ 
ted  courses  the  software  quality  assurance  staff  will  be 
able  to: 

•  Assist  in  determining  the  needs  of  the  buying 
activity; 

•  Evaluate  the  general  approach  to  the  system 
development ; 

•  Analyze  requirements  to  determine  and  report 
conf 1 icts ; 

•  Evaluate  a  software  quality  assurance  plan 
against  constraints  imposed  by  cost,  schedule, 
and  operating  environment; 

•  Track  corrective  action  procedures; 

•  Communicate  effectively  with  supervisors,  subor¬ 
dinates,  contractor  staff,  and  counterparts; 

•  Interact  effectively  with  others  involved  in 
other  disciplines  (for  example,  hardware  and 
budget)  who  are  associated  with  the  project. 

The  greatest  sense  of  purpose  for  software  quality 
assurance  personnel  will  come  from  the  knowledge  that  the 
headquarters:  (1)  Is  aware  of  the  importance  of  software 

quality  assurance,  (2)  Is  cognizant  that  staff  people  have 
the  desire  to  perform  their  functions  as  expertly  as  pos¬ 
sible  and  are  willing  to  be  trained  to  that  end,  and  (3)  Is 
willing  to  make  the  necessary  provisions.  Confidence  in 
one's  ability  and  appreciation  of  the  significance  of  one's 
professional  performance  can  often  overcome  the  frustrations 
surrounding  daily  routine. 
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SECTION  V 


DETAILED  AREAS  FOR  IMPROVEMENT 


5.1  OVERVIEW 

Most  respondents  stated  that  the  original  requirements  for 
software  quality  assurance  are  often  de -emphasized  during  the 
software  development  projects.  When  cost  overruns  are  antici¬ 
pated  there  is  the  tendency  to  "cut  corners"  in  the  area  of 
software  quality  assurance.  The  software  quality  assurance 
management  plan  and  life  cycle  model  plan  must  be  well  struc¬ 
tured;  software  quality  assurance  activities  must  be  appropri¬ 
ately  integrated  into  the  life  cycle  plan;  software  quality 
assurance  tools  and  techniques  must  be  independently  planned 
and  used;  and  lines  of  communication  must  be  firmly  established. 

If  software  quality  assurance  requirements  are  specified  in  RFP's, 
and  plans  are  developed  for  fulfilling  these  requirements,  the 
selection  of  software  quality  assurance  as  the  area  for  de-empha¬ 
sis  will  be  made  more  difficult  if  not  impossible.  Cost  overruns 
should  be  greatly  reduced  and  the  need  to  "cut  corners"  will  be 
minimized. 

5.2  CRITICAL  AREAS  FOR  IMPROVEMENT 

As  a  result  of  the  analysis  of  current  software  quality  as¬ 
surance  practices  and  the  evaluation  of  requested  changes,  four 
areas  for  improvement  are  determined: 

(1)  Software  Quality  Assurance  Guidance  Documents, 

(2)  The  Relation  of  Software  Quality  Assurance  to  Software 
Deve lopment , 

(3)  Communication  Methods  and  Model  Documentation,  and 

(4)  Training  and  Staffing. 


In  the  following  subsections,  specific  suggestions  for  im¬ 
provement  are  detailed.  In  most  cases,  areas  for  improvement 
encompass  all  organizations,  namely, the  Defense  Logistics  Agency, 
the  Electronic  Systems  Division  and  The  Air  Force  Contract  Manage¬ 
ment  Division.  Where  a  particular  improvement  is  applicable  to  a 
specific  organization,  this  has  been  so  specified. 

5.3  AREAS  FOR  IMPROVEMENT  IN  SOFTWARE  QUALITY  ASSURANCE 
GUIDANCE  DOCUMENTS 

If  the  quality  of  software  is  to  reflect  improvement  several 
concepts  must  be  recognized  and  accepted.  First,  although  soft¬ 
ware  development  and  software  quality  assurance  are  considered 
to  be  separate  unique  disciplines,  they  must  be  considered  as 
interdependent  when  developing  plans  and  procedures. 

Second, a  software  quality  assurance  program  can  be  broken 
into  three  phases:  (1)  planning,  (2)  development,  and  (3)  imple¬ 
mentation.  While  the  boundaries  of  these  three  phases  are  not 
distinct,  it  is  necessary  to  accomplish  all  three  to  have  an  ef¬ 
fective  software  quality  assurance  program.  The  major  product 
of  this  first  phase  is  the  software  quality  assurance  management 
plan.  Phases  two  and  three  are  discussed  in  subsections  5.4  and 
5.5. 

5.3.1  Criteria  for  Improvement 

In  order  to  develop  a  software  quality  assurance 
management  plan,  guidance  documents  must  specify  guide¬ 
lines  for  all  the  major  elements  of  a  plan.  With  the  aid 
of  these  guidelines,  software  quality  assurance  management 
plans  can  be  developed  that  are  specific  to  a  project.  The 
major  elements  that  should  be  included  in  a  software  quality 
assurance  guidance  document  are: 

• 

I 
I 
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Precontract  planning  to  incorporate  software  quality 
assurance  provisions  into  request  for  proposals; 


•  Definition  of  software  quality  assurance  tasks  and 
methods  ; 

•  Division  of  organizational  responsibilities  for  the 
software  quality  assurance  tasks; 

•  Development  of  the  basic  software  quality  assurance 
communication  structure;  and 

•  Selection  of  personnel  for  carrying  out  all  tasks. 

5.3.2  Areas  for  Improvement 

In  the  Defense  Logistics  Agency,  the  primary  guidance 
document  for  software  quality  assurance  is  DLAM  8200.1, 
Section  IX,  Part  15,  Procurement  Quality  Assurance  for  Com¬ 
puter  Software.  Although  this  document  addresses  some  of 
the  basic  elements  of  a  software  quality  assurance  manage¬ 
ment  plan,  procedures  for  implementation  are  heavily  hard¬ 
ware  oriented.  For  example,  methods  for  performing  Proce¬ 
dures  Revxew  and  Procedures  Evaluation  are  referenced  to 
those  areas  in  the  manual  that  wore  developed  for  hardware 
quality  assurance. 

Within  the  Electronic  Systems  Division,  the  contrac¬ 
tor's  Computer  Program  Development  Plan  in  conjunction  with 
AFR  800-14,  Acquisition  and  Support  Procedures  for  Computer 
Resources  in  Systems,  and  MIL-STD-485,  Configuration  Manage¬ 
ment  Practices  for  Systems,  Equipment,  Munitions,  and  Com¬ 
puter  Programs,  are  the  primary  guidance  documents  for  soft¬ 
ware  development.  There  is  no  document  solely  for  software 
quality  assurance. 

The  Electronic  Systems  Division's  Software  Acquisition 
Management  Guidebooks,  in  particular,  ESD-TR-  7 7  -  22  5  ,  Software 
Quality  Assurance  arc  excellent  plans,  yet,  little  or  no  men¬ 
tion  was  made  of  them  by  survey  respondents.  This  may  be 
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because  they  are  guidebooks  and  not  mandatory  regulations. 
Wider  use  of  these  documents  with  some  modifications  is  en¬ 
couraged. 

In  the  Air  Force  Contract  Management  Division,  the 
primary  guidance  document  for  software  quality  assurance  is 
AFCMDR  74-1,  Procurement  Quality  Assurance  Program,  Chapter 
15.  Chapter  15  does  deal  directly  with  software  quality  as¬ 
surance,  but  references  to  other  guidelines  that  are  hard¬ 
ware  oriented  are  used  extensively.  Expansion  of  all  docu¬ 
ments  dealing  with  software  quality  assurance  will  present 
a  more  cohesive  structure  for  evaluators  to  follow  and  will 
provide  the  tools  necessary  for  the  accomplishment  of  the 
plan.  AFCMDR  800-1,  Contract  Management  Engineering,  and 
AFCMDP  800-2,  Contract  Management  Engineering  Guide,  provide 
guidance  for  performing  contract  management  engineering  func¬ 
tions  in  support  of  the  Air  Force  Contract  Management  Divi¬ 
sion  (AFCMD)  mission. 

AFCMDR  178-1,  Contractor  Management  System  Evaluation 
Program,  is  a  well  organized,  structured  method  for  evaluating 
contractor  plans.  However,  there  is  no  section  dealing 
with  software  quality  assurance.  Adding  this  section  will 
provide  software  quality  assurance  personnel  with  detailed 
questions  specific  to  software  which  would  likely  be  omitted 
if  reliance  is  placed  solely  on  the  Quality  Assurance  Section. 

After  a  review  of  all  guidance  documents  for  the  major 
elements  of  a  software  quality  assurance  management  plan  the 
following  areas  for  improvement  are  determined  for  the  Defense 
Logistics  Agency,  the  Electronic  Systems  Division  and  the  Air 
Force  Contract  Management  Division: 

•  Greater  involvement  of  software  quality  assurance  per¬ 
sonnel  in  early  acquisition  phases; 
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•  Specification  of  division  of  functions  in  Letters 
of  Delegation  and  Memoranda  of  Agreement; 

•  Expansion  of  regulations  to  detail  government  software 
quality  assurance  procedures  for  monitoring  contrac¬ 
tors;  and 

•  Development  of  review  and  evaluation  procedures  speci¬ 
fically  software  quality  assurance  oriented. 

5.3.2. 1  Greater  Involvement  of  Software  Quality 
Assurance  Personnel  in  Early  Acquisition 
Phases 

Involvement  of  software  quality  assurance 
personnel  early  in  the  acquisition  cycle  is  imperative. 
Based  on  their  past  experiences,  software  quality  as¬ 
surance  personnel  are  aware  of  areas  where  software 
quality  assurance  was  de-emphasized  during  software 
development.  If  they  participate  in  request  for  pro¬ 
posal  preparation,  they  will  recommend  incorporation  of 
software  quality  assurance  requirements.  As  a  result, 
contractors  will  submit  clearly  defined  plans  and  pro¬ 
cedures.  At  the  time  of  contract  award  negotiations, 
clear  definitions  of  what  is  expected  from  the  contrac¬ 
tor's  and  the  government's  quality  assurance  programs 
will  be  established.  Later  problems  causing  cost  over¬ 
runs  and  schedule  delays  will  be  minimal. 

The  Electronic  Systems  Division's  So  ft ware 
Acquisition  Management  Guidebooks  detail  the  procedures 
of  REP  preparation,  proposal  evaluations  and  contract 
award  negotiations.  The  inclusion  of  software  quality 
assurance  personnel  as  participants  in  these  activities 
is  not  so  specified.  Software  quality  assurance  partici¬ 
pation  would  require  a  minimum  effort  and  yield  signifi¬ 
cant  benefits. 
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In  summary,  incorporation  of  a  pre-planning 
section  in  the  software  quality  assurance  guidance 
documents  will  result  in  the  following  benefits: 

•  Specifications  on  software  quality  assurance 
requirements  in  requests  for  proposals  and  con¬ 
tracts  , 

•  Inclusion  of  software  quality  assurance  require¬ 
ments  in  requests  for  proposals  and  contracts, 

•  Reduction  of  future  problems  through  utilization 
of  past  experiences, 

•  Reduction  of  delays  in  schedule  performance,  and 

•  Reduction  in  costs. 

5. 3. 2. 2  Clear  Division  of  Functions 

Where  unique  quality  assurance  actions  to  \ 

be  performed  by  the  Contract  Administration  Office 
are  specified  through  quality  assurance  Letters  of 
Delegation  or  Memoranda  of  Agreement,  respondents 
were  able  to  supply  very  little  data.  Respondents  i 

seemed  to  have  little  knowledge  of  how  division  of 
functions  are  negotiated.  However,  respondents  stated 
that  schedule  slippages,  cost  overruns  and  a  de-em-  1 

phasis  of  software  quality  assurance  requirements  occur, 
generally,  because  of  poor  planning  and  lack  of  govern¬ 
ment  technical  expertise. 

Regulations  governing  the  division  of  functions 
do  exist;  therefore,  a  breakdown  in  communications  can  be 
assumed  to  exist.  These  functions  are  delegated  by  the 
Program  Office.  Negotiations  ensure  where  resources 
are  available  to  perform  special  tasks.  Only  those  re' 
spondents  dealing  with  the  everyday  activities  of  software 


quality  assurance  are  aware  of  available  resources. 

For  negotiations  to  be  effective,  input  from  the  soft¬ 
ware  quality  a  surance  staff,  both  at  management  and 
operations  levels,  is  essential.  This  will  result  in 
developing  Memoranda  of  Agreement  with  more  detailed 
specifications  regarding  the  provisions  on  software. 
Memoranda  of  Agreement  and  Letters  of  Delegation  that 
are  written  early  in  the  acquisition  process  with  in¬ 
put  from  the  above  mentioned  levels  will  clearly  define 
responsibilities  and  communicate  these  responsibilities 
to  all  concerned. 

In  summary,  Letters  of  Delegation  and  Memor¬ 
anda  of  Agreement  written  based  on  communication  be¬ 
tween  the  Program  Office  and  Contract  Administration 
Office  will: 

•  Specify  provisions  on  software, 

•  Determine  availablity  of  technical  resources, 

•  Assign  tasks  as  they  will  occur  during  the  soft¬ 
ware  life  cycle  phases,  and 

•  Establish  lines  of  communication. 

5. 3. 2. 3  Expansion  of  Regulations  to  Detail  Software 
Quality  Assurance  Procedures^  for  Monitoring 
Contractors  ~  "  ””  ~ 

Regulations  and  standards  referenced  in  exist¬ 
ing  software  quality  assurance  guidance  documents  are 
written  in  general  terms.  For  example,  MIL-S-52799A, 
Software  Quality  Assurance  Program  Requirements,  lists 
contractor  plans  that  the  software  quality  assurance 
personnel  must  identify  as  existing  and  consistent 
with  contractual  requirements,  but  there  are  no  details 
provided  to  ensure  that  the  requirements  are  adequate. 


In  some  cases,  references  to  other  regulations  and 
standards  are  extensive  and  cross  referencing  can  be 
considerably  time  consuming.  Again,  many  regulations 
and  standards  are  based  on  procedures  developed  for 
hardware  quality  assurance. 

Expansion  of  software  quality  assurance 
guidance  documents  should  include  consolidation  of 
relevant  information  from  all  necessary  regulations 
into  one  general  document.  Based  on  this  general 
plan,  plans  tailor  made  for  specific  contracts  can 
be  formulated  without  repetition  or  standard  require¬ 
ments.  Regulations  for  software  quality  assurance 
should  be  unique  regulations  geared  specifically  for 
software  as  should  the  software  quality  assurance 
guidance  documents. 

5. 3. 2. 4  Development  of  Review  and  Evaluation 

Procedures  that  are  Speci fTcal ly~Sortware 
Quality  Assurance  Oriented 

References  to  hardware  oriented  procedures 
for  evaluating  software  quality  assurance  should  be 
eliminated  from  software  quality  assurance  guidance 
documents.  There  are  few  similarities  between  the 
tangible  hardware  equipment  and  the  software  products. 
There  are  also  few  similarities  between  evaluating 
hardware  and  evaluating  software  products.  Again, 
software  quality  assurance  evaluation  procedures 
should  be  unique  procedures. 

5.4  AREAS  FOR  IMPROVEMENT  IN  THE  RELATION  OF  SOFTWARE  QUALITY 

ASSURANCE" TO  S0FTWARE~^EVE10'MET7T - - 

Integrating  software  quality  assurance  activities  with  the 
software  development  activities  entails  setting  up  the  working 
relationship  between  functional  organizations,  and  documenting 
these  interfaces  with  operating  procedures  and  instructions. 


72 


A  system  of  assesments,  evaluations  and  reviews,  both  formal 
and  informal,  are  planned  for  specific  milestones,  baseline  docu¬ 
mentation  and  other  documentation  required  in  the  development 
process . 

5.4.1  Criteria  for  Improvement 

Specifically,  some  of  the  major  responsibilities  of 
the  software  quality  assurance  group  consisting  of  program 
office  and  CAO  representatives  are: 

•  Ensuring  that  sound  software  development  practices  are 
followed  throughout  the  software  development  life  cycle, 

•  Ensuring  that  software  quality  control  procedures 
required  by  contract  are  planned  by  the  contractor, 

•  Developing  tools  and  techniques  for  monitoring  con¬ 
tractor  reviews  and  audits,  and 

•  Reviewing  all  acceptance  test  plans  and  procedures. 

In  the  development  of  monitoring  and  evaluation  tools 
and  techniques,  software  quality  assurance  personnel  should 
develop  standard  independent  tools  that  can  be  adapted  to 
specific  based  contract  requirements  and  contractor's  plans. 
Reliance  should  not  be  placed  solely  on  contractor's  results 
of  reviews. 

5.4.2  Areas  For  Improvement 

Based  on  a  review  of  life  cycle  models,  software 
development  phases  and  related  software  quality  assurance 
activities,  the  following  areas  of  improvement  were  determined 
for  the  Defense  Logistics  Agency,  the  Electronic  Systems 
Division  and  the  Air  Force  Contract  Management  Division: 

•  Integration  of  software  quality  assurance  activities 
into  the  total  life  cycle  development  model, 

•  Participation  in  establishing  and  evaluating  baselines, 

•  Development  of  standard  formats  for  writing  documents, 
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•  Development  of  specific  tools,  techniques  and  methodol¬ 
ogies  for  monitoring  and  evaluating  contractor  soft- 
ware  quality  assurance  plans  and  procedures  that  are 
required  by  contract. 

5. 4. 2.1  Integration  of  Software  Quality  Assurance 

Activities  into  Life  Cycle  Development  Model 

Since  DLA  is  in  the  process  of  revising  its 
life  cycle  model,  comments  concerning  the  present  model 
are  limited.  DLAM  8200.1,  Procurement  Quality  Assur¬ 
ance  exhibits  charts  of  quality  control  documents 
and  the  typical  sequence  of  events  for  computer  soft¬ 
ware  programs  with  quality  assurance  concerns  speci¬ 
fied.  Incorporation  of  the  data  from  these  charts 
into  the  revised  life  cycle  model  would  create  a 
single  reference  point  which  would  facilitate  plan¬ 
ning.  Modifications  could  be  made  on  a  contract  basis 
and  special  requirements  and  activities  could  be  inte¬ 
grated  with  minimum  difficulty. 

Electronic  Systems  Division's  Software 
Acquisition  Management  Guidebook  Series  forms  a  solid 
working  basis  from  which  to  develop  a  life  cycle 
model  with  integrated  quality  assurance  activities. 

The  guidebooks  develop  in  great  detail  the  quality 
assurance  activities  that  are  to  be  performed  during 
each  phase,  corresponding  documentation  to  be  re¬ 
viewed  and  documentation  to  be  completed.  What 
remains  is  to  highlight  and  consolidate  in  one 
graphic  depiction  for  each  life  cycle  phase,  the 
major  software  quality  assurance  activities  per¬ 
formed,  and  standard  documentation  involved.  This 
type  of  model  will  facilitate  planning.  With  this 
as  a  basis,  modifications  needed  to  adhere  to  special 
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requirements  of  particular  contracts  can  be  made  with 
minimum  difficulty. 


The  Air  Force  Contract  Management  Division's 
Procurement  Quality  Assurance  Program,  AFCMDR  74-1, 
Chapter  IS,  illustrates  several  charts  of  life  cycle 
model  phases  with  contractor  requirements  and  products 
specified.  AFCMDR  800-1,  Contract  Management  Engineer¬ 
ing  and  AFCMDP  800-2,  Contract  Management  Engineering 
Guide  also  detail  the  breakdown  of  life  cycle  phases 
and  the  activities  to  be  performed  by  AFCMD  Engineer¬ 
ing.  These  documents  form  a  solid  basis  where  quality 
assurance  activities  can  be  integrated  without  the 
necessity  of  a  major  revision. 

Integrating  quality  assurance  activities  into 
a  life  cycle  model  can  be  accomplished  with  a  minimum 
of  effort  by  the  Defense  Logistics  Agency,  the  Elec¬ 
tronic  Systems  Division  and  the  Air  Force  Contract 
Management  Division.  Since  basic  data  exists  in  docu¬ 
mentation  of  these  three  organizations  regarding  soft¬ 
ware  quality  assurancq  functions,  all  that  is  required 
is  to  select  software  quality  assurance  functions  and 
map  them  into  the  appropriate  phases  of  the  life  cycle 
model  to  present  an  organized  visual  representation  of 
the  software  development  process. 

5. 4. 2. 2  Participation  in  Establishing  and  Evaluating 

Baselines 

A  baseline  is  a  reference  point  which  estab¬ 
lishes  a  basis  of  understanding  among  all  project  staff. 
Software  quality  assurance  personnel  within  the  Defense 
Logistics  Agency,  the  Electronic  Systems  Division  and 
the  Air  Force  Contract  Management  Division  do  work  with 
baselines.  The  Electronic  Systems  Division  sometimes 
participates  in  the  development  of  baselines.  The 
Defense  Logistics  Agency  and  the  Air  Force  Contract 


Management  Division  do  not.  Not  only  should  software 
quality  assurance  personnel  use  baselines  as  refer¬ 
ence  points  but  they  should  be  involved  in  their 
development.  At  the  very  least,  they  should  eval¬ 
uate  baseline  documentation. 

According  to  MIL-STD-483  baselines  are  de¬ 
fined  as: 

•  Functional  Baseline  -  the  initial  approved 
functional  configuration  identification; 

•  Allocated  Baseline  -  the  initial  approved 
allocated  configuration  identification;  and 

•  Product  Baseline  -  The  initial,  approved  or 
conditionally  approved  product  configuration 
identification . 

Software  quality  assurance  personnel  should 
evaluate  at  a  minimum: 

•  Functional  baseline  documents  to  determine  if 
the  system  will  meet  the  requirements; 

•  Allocated  baseline  documents  to  determine  if 
the  functions  to  be  performed  by  the  software 
components  are  complete  and  ready  for  use  in 
building  and  integrating  the  system;  and 

•  Product  baseline  documents  to  ensure  that 
corrections  to  deficiencies  found  in  the 
testing  of  the  product  baseline  have  been 
documented  and  contractual  requirements  are 
being  met. 

Baselines  are  the  principal  communications 
media  between  the  developer,  the  Contract  Administra¬ 
tion  Offices  and  the  acquisition  activity.  To  exclude 
software  quality  assurance  from  the  evaluation  of 
baseline  documentation  is  to  exclude  the  essential 
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control  system  and,  therefore,  excludes  or  at  least 
diminishes  considerably  the  probability  of  quality 
software . 

5. 4. 2. 3  Development  of  Model  Documents 

Quality  assurance  as  a  discipline  is  com¬ 
monly  invoked  throughout  governmental  and  industrial 
organizations  with  reasonable  standardization  when 
applied  to  systems  comprised  only  of  hardware.  But 
there  is  enormous  variation  in  thinking  and  practice 
when  the  quality  assurance  discipline  is  invoked 
for  software  development  or  for  a  system  containing 
software  components.  Software,  as  a  form  of  informa¬ 
tion,  cannot  be  standardized;  only  structures  for 
defining  and  documenting  software  can  be  standardized. 
If  the  structures  for  all  software  documentation  have 
been  clearly  defined  from  the  early  stages  of  system 
acquisition,  specific  formats  for  documents  will  be 
available  prior  to  initiation  of  the  attendant  devel¬ 
opment  phase.  Considerable  manpower  can  be  saved  if 
the  engineers  know  exactly  what  they  must  produce  in 
the  way  of  documentation  and  exactly  how  and  when. 

Although  Data  Item  Descriptions  (DIDs)  and 
Contract  Data  Requirements  Lists  (CDRLs)  do  define 
the  requirements  for  documentation,  respondents  noted 
that  documents  are  sometimes  poorly  written  and  very 
often,  behind  schedule.  Documents  may  contain  what 
they  are  supposed  to  contain  and  still  be  inferior. 
Preparation  of  model  document  packages  can  range  from 
simple  outlines  structuring  the  contents  into  some 
logical  order,  to  preparation  of  complete  samples  of 
each  major  type  of  document  that  is  generally 
produced . 

If  specific  formats  for  superior  documenta¬ 
tion  are  standardized  and  required,  and  an  initial 
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period  of  implementation  and  enforcement  is  allowed 
in  order  to  work  out  problems,  documentation  from  con¬ 
tractors  will  become  more  uniform.  In  fact,  it  will 
be  easier  for  them  to  prepare.  The  reviews  performed 
by  software  quality  assurance  will  be  performed  with 
greater  facility  and  the  quality  of  software  will 
improve  within  schedule  and  cost  constraints.  Rigi¬ 
dity  is  not  being  suggested  here.  What  is  being 
suggested  is  a  basic  format  which  will  allow  tailor¬ 
ing  for  specific  project  requirements. 

5. 4. 2. 4  Development  of  Tools,  Techniques  and 

Methodologies 

Special  tools,  techniques,  and  methodologies 
are  planned,  controlled,  and  applied  to  support  soft¬ 
ware  quality  assurance  objectives  and  assist  in 
software  verification.  On  the  assumption  that  the 
software  life  cycle  model  has  been  planned  and  con¬ 
structed  as  previously  discussed,  reviews  and  audits 
are  then  seen  as  an  integral  part.  Due  to  the  com¬ 
plexity  and  diversity  of  software  projects,  it  is 
not  possible  nor  reasonable  to  suggest  which  parti¬ 
cular  tools  to  use.  However,  some  comments  on  tool 
development  are  appropriate. 

The  government's  primary  tools  for  monit¬ 
oring  contractors  are  checklists  for  assessing  con¬ 
tractor  documentation.  If  documentation  is  planned, 
associated  reviews  and  audits  will  discover  errors 
early  on  in  each  phase,  provide  for  correction,  save 
time  and  costs  and  allow  a  project  to  remain  on 
schedule.  Audits  will  be  conducted  as  official 
audits  to  evaluate  conformance  of  prepared  documen¬ 
tation  to  contractual  requirements.  Once  the  require¬ 
ments  of  each  phase  have  been  specified  with  its 
corresponding  reviews,  audits  and  documentation, 
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checklists  for  assessment  of  quality  can  and  should 
be  tailor  made  to  the  contract. 

Checklists  should  be  independently  devel¬ 
oped  by  software  quality  assurance  personnel  based 
on  contractor  Computer  Program  Development  Plans, 
and  contract  requirements.  If  requirements  for  a 
software  configuration  management  plan  have  been 
specified,  checklists  should  be  constructed  based 
on  those  requirements.  It  is  possible  to  construct 
a  standard  checklist  which  includes  the  elements  of 
a  good  software  configuration  management  plan.  This 
standard  checklist  can  then  be  used  to  evaluate 
contractors'  plans  for  compliance  with  contract 
requirements,  procedures  for  implementation  and, 
most  importantly,  the  adequacy  of  their  procedures. 

According  to  the  Electronic  Systems  Division, 
checklists  are  modeled  after  the  DIDs  and  standards. 

If  this  is  done  according  to  the  above  principles  and 
are  adaptable  to  all  projects,  then  the  software 
quality  assurance  staff's  tasks  of  monitoring  and 
evaluating  should  be  less  tedious. 

SAI  was  able  to  obtain  very  few  checklists, 
primarily  due  to  the  proprietary  data  involved, 
therefore,  a  checklist  evaluation  was  not  possible. 
However,  many  respondents  indicated  that  they  do  use 
checklists  and  complete  them  based  on  "what  the  con¬ 
tractor  says  has  been  done”.  The  criteria  is  the 
good  rapport  developed  between  the  contractor  and 
government  software  quality  assurance  personnel. 

Witnessing  of  software  testing  should  follow 
the  same  plan.  Requirements  should  be  documented  and 
checklists  developed  from  the  requirements.  Software 
quality  assurance  personnel  should  know  the  test 
sequence,  the  test  input  data,  the  data  base  and  the 
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software  configuration  in  order  to  identify  irregular 
ities  in  their  evaluations.  Observations  of  the  test 
itself  and  evaluation  of  the  test  output  data  con¬ 
stitute  the  basis  on  which  it  is  determined  if  the 
test  objectives  have  been  met,  the  pertinent  require¬ 
ments  verified  and  the  acceptance  criteria  satisfied. 
Government  software  quality  assurance  evaluators 
should  be  able  to  evaluate  a  test  report  in  light  of 
the  objectives  of  the  test  and  note  any  anomalies. 
Disciplined  control  of  the  testing  effort  should  be 
maintained  by  emphasizing  comprehensive  and  precise 
definition  of  test  plans,  and  evaluation  of  test 
achievement  against  the  test  plan  at  periodic  check¬ 
points  . 

For  complex  programs,  automated  tools  are 
becoming  a  necessity  in  order  to  permit  adherence  to 
system  requirements.  Use  of  automated  tools  may  make 
the  software  quality  assurance  evaluator's  task  less 
tedious  but  he  or  she  will  need  to  be  educated  as  to 
their  purpose  and  applicability  to  specific  projects. 
They  will  need  to  be  aware  of  the  appropriateness  of 
the  tool  in  the  life  cycle  development  phase,  what 
the  output  should  indicate,  and  how  it  interacts  with 
the  activities  of  other  phases. 

Reviewing,  monitoring  and  witnessing  of  con¬ 
tractor's  quality  assurance  procedures  are  satisfac¬ 
tory  if  the  technical  expertise  exists  to  perform 
these  activities.  Increased  technical  skill  will 
allow  software  quality  assurance  personnel  to  partici 
pate  actively  in  these  functions.  The  purpose  is  not 
to  dictate  to  the  contractor  how  to  perform  tasks  , 
but  to  provide  the  software  quality  assurance  evalu¬ 
ator  with  the  basic  understanding  of  the  techniques 
used  by  the  contractor.  Checklists  can  then  be  com¬ 
pleted  from  a  technical  evaluation  rather  than,  again 
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simply  determining  that  a  contractor  has  or  has  not 
addressed  a  quality  requirement. 

5.5  AREAS  FOR  IMPROVEMENT  IN  COMMUNICATION  METHODS  AND  MODEL 

DOCUMENTS 

Software  quality  assurance  plans,  no  matter  how  well  struc¬ 
tured,  will  fall  short  of  their  objectives  if  the  system  of 
communication  is  not  well  planned.  In  order  for  any  program  to 
achieve  its  objectives,  all  project  personnel  must  be  aware  of 
the  objectives,  the  procedures  to  be  used  to  obtain  the  objectives, 
and  the  methods  and  lines  of  communications  to  be  used.  Communi¬ 
cation  is  defined  as  the  transmission  of  meaning  to  others.  Formal 
communication  emphasizes  the  dissemination  of  information  and 
informal  communication  consists  of  exchanges  between  two  or  more 
people.  Both  are  necessary  and  though  the  latter  is  more  loosely 
constructed,  some  guidelines  are  necessary  to  govern  its  use. 

5.5.1  Criteria  for  Improvement 

The  following  guides  are  presented  to  aid  in  devel¬ 
oping  effective  communication: 

•  Determine  the  purpose  of  the  communication; 

•  Develop  a  system  of  communication  flow  to  be  followed 
during  project  implementation; 

•  Determine  appropriate  standards  and  regulations; 

•  Ensure  that  all  project  personnel  are  made  aware  of 
the  communication  system. 

The  communication  system  need  not  be  rigid.  It  should 
maintain  a  degree  of  flexibility,  yet  ensure  that  all  per¬ 
sonnel  send  and  receive  information  relevant  to  their 
functions . 

5.5.2  Areas  for  Improvement 

Based  on  the  criteria  for  improvement  of  communication 
methods,  the  following  areas  of  improvement  are  determined 
for  the  Defense  Logistics  Agency,  the  Eletronic  Systems 
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Division  and  the  Air  Force  Contract  Management  Division: 

•  Determine  the  formal  and  informal  methods  of 
communication ; 

•  Determine  the  flow  of  communication; 

•  Improve  regulations  and  specifications  that  regulate 
communication  methods;  and 

•  Ensure  that  all  project  personnel  receive  information 
that  affects  their  functions. 

5. 5. 2.1  Determine  Formal  and  Informal  Methods 

of  Communication 

Letters ,  memos  and  reports  that  are  to  be  a 
matter  of  record  should  be  planned  before  the  start 
of  a  project.  The  subject  of  these  types  of  communi¬ 
cation,  for  example,  requests  for  changes,  corrective 
action  reports,  performance  assessments,  acceptance 
reports,  etc.,  should  be  formatteu  and  prepared  for 
use.  The  exact  information  that  is  to  be  included 
should  be  determined  in  order  that  uniformity  be 
preserved. 

Board  meetings  and  panel  discussions  should 
also  be  planned  before  the  project  begins.  The  pur¬ 
pose  should  be  clearly  stated  and  the  frequency 
determined.  If  one  or  more  of  the  above  mentioned 
reports  is  the  subject  of  the  meeting,  all  partici¬ 
pants  should  be  informed.  If  software  quality  as¬ 
surance  matters  are  affected  then  software  quality 
assurance  personnel  should  participate.  This  is  not 
always  the  case  according  to  respondents. 

Most  respondents  stated  that  informal  methods 
of  communication  (for  example,  telephoning  for  advice 
or  assistance)  are  adequate  and  are  effective.  In 
establishing  a  communication  system,  this  should  not 
be  lost. 
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5 . 5 . 2 . 2  Determine  the  Flow  of  Communication 

A  common  complaint  among  respondents  was 
that  decisions  are  made  concerning  software  quality 
assurance  and  software  quality  assurance  personnel 
are  not  informed.  Again,  it  should  be  a  matter  of 
record  as  to  who  should  receive  pertinent  information 
and  a  distribution  system  should  be  established.  If 
all  project  personnel  have  a  list  of  all  other  project 
staff,  there  will  be  less  opportunity  for  a  staff 
member  to  be  omitted  in  the  distribution.  If  tasks 
have  been  specifically  delegated,  managers  will  direct 
their  communications  to  the  appropriate  staff  members, 
and  staff  members  will  submit  reports  to  all  who 
should  receive  them.  A  tracking  system  is  relatively 
easy  to  establish,  saves  time,  and  avoids  the  frustra¬ 
tions  involved  in  not  receiving  communications. 

5 . 5 . 2 . 3  Improve  Regulations  and  Specifications 

Another  common  complaint  from  respondents 
was  that  software  quality  assurance  personnel  are 
not  made  aware  of  decisions  made  and  tasks  delegated 
through  Memoranda  of  Agreement  and  Letters  of  Delega¬ 
tion.  Regulations  detailing  the  writing  of  MOAs/LODs 
should  emphasize  coordination  between  the  Program 
Office  and  the  software  quality  assurance  staff  before 
decisions  are  finalized.  This  necessitates  that 
specific  provisions  on  software  be  included.  When 
modifications  are  necessary,  renegotiations  between 
the  Program  Office  and  software  quality  assurance 
staff  are  necessary  to  ensure  that  software  quality 
assurance  is  not  de -emphasized  and  new  tasks  assigned 
fall  within  the  available  technical  expertise  or  can 
be  acquired  easily  and  quickly. 
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5. 5. 2. 4  Orient  Project  Personnel  to 
Communication  System 

In  order  that  all  project  personnel  be  made 
aware  of  the  communication  system,  a  period  of 
orientation  is  required.  Regulations  should  specify 
the  basic  elements  to  be  included  in  a  communication 
system.  However,  each  organization  and  project  must 
apply  regulations  in  a  manner  peculiar  to  its  own 
structure  and  needs.  New  personnel  must  be  made 
aware  of  the  communication  distribution  system,  the 
types  of  records  to  be  maintained,  standard  forms 
required,  and  special  formats  to  be  followed.  Points 
of  contact  can  be  established  for  those  occasions 
when  special  technical  assistance  is  needed.  If  the 
existing  communication  system  is  re-structured  for 
improvement  then  all  personnel  will  require  reorienta¬ 
tion.  In  restructuring  the  communication  system  it  is 
important  to  allow  for  a  period  of  adjustment. 

5.6  TRAINING  AND  STAFFING 

As  has  been  stated,  no  software  quality  assurance  management 
plan,  no  software  development  plan  and  no  communication  system 
however  well  planned  and  constructed  will  by  itself  assure  the 
quality  of  software.  The  key  factor  is  professionally  trained 
personnel  who  are  convinced  of  the  vital  importance  of  software 
quality  assurance  and  its  place  in  software  development. 

5.6.1  Criteria  for  Improvement 

Performing  software  quality  assurance  is  almost 
impossible  without  knowledge  of  software  and  its  develop¬ 
mental  processes.  In  order  for  software  quality  assurance 
personnel  to  perform  their  functions  expertly  and  efficiently 
they  must  have  the  technical  capability  to: 

•  Understand  software  requirements,  design,  code  and 
test  procedures, 
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•  Interact  with  contractor  software  quality  assurance 
staff  and  software  engineers,  and 

•  Independently  evaluate  the  contractor's  plans  for 
assuring  the  quality  of  software. 

Software  quality  assurance  managers  must  be  aware  of 
the  required  technilogy  and  possess  the  capabilities  and 
skills  necessary  to: 

•  Manage  personnel  on  various  levels,  and 

•  Maintain  communication  flow  as  planned. 

In  addition  software  quality  assurance  personnel  must 
have  the  ability  to  make  value  judgments  in  unforeseen  cir¬ 
cumstances  requiring  immediate  attention.  They  must  preserve 
a  strict  control  over  plans  that  have  been  rigidly  organized, 
yet  be  knowledgeable  enough  to  determine  where  flexible 
decisions  are  warranted  and  be  prepared  to  make  them.  This 
requires  a  firm  foundation  in  the  principles  of  management 
and  in  the  software  development  process. 

5.6.2  Areas  for  Improvement 

The  Defense  Logistics  Agency,  the  Electronic  Systems 
Division  and  the  Air  Force  Contract  Management  Division  have 
personnel  who  have  been  exposed  to  some  software  courses. 
However,  these  appear  to  have  been  obtained  on  an  as  needed 
basis  and  in  isolated  situations.  To  assign  personnel  to  a 
project  and  then  provide  training  is  cost  ineffective. 
Projects  continue  and  training  received  may  be  too  late  to 
affect  the  quality  of  the  software  being  developed.  The 
following  areas  for  improvement  in  training  and  staffing 
are  determined  for  the  Defense  Logistics  Agency,  the  Elec¬ 
tronic  Systems  Division  and  the  Air  Force  Contract  Manage¬ 
ment  Division. 

Personnel  must  be  trained  in  a  structured  manner  in 
the  following  skills; 

•  Management  of  software  quality  assurance  projects; 
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•  Software  development  processes  with  software  quality 
assurance  procedures  integrated; 

•  Use  of  software  development  tools  and  techniques; 

•  Procedures  for  developing  software  quality  assurance 
checklists . 

In  addition,  personnel  assigned  to  projects  should 

receive  orientation  regarding: 

•  Organizational  responsibilities  and  operations  within 

the  Defense  Logistics  Agency,  the  Electronic  Systems 
Division  and  the  Air  Force  Contract  Management  Division 

•  Contractors'  facilities  and  methods  of  operations; 

•  Personnel  responsibilities;  and 

•  Lines  of  communication. 
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SECTION  VI 


RECOMMENDATIONS 


6.1  OVERVIEW 

Identifying  those  Areas  for  Improvement  in  Section  V  with 
the  greatest  impact  on  quality  software,  SAI  recommends  changes 
in  four  areas:  (1)  Government  Software  Quality  Assurance  Guidance 
Documents,  (2)  The  Relation  of  Software  Quality  Assurance  to  Soft¬ 
ware  Development,  (3)  Communication  Methods  and  Model  Documents, 
and  (4)  Training  and  Staffing.  Improvements  in  tools,  techniques 
and  communication  methods  are  predicated  upon  software  quality 
assurance  guidance  documents  that  recognize  the  development  of 
software  as  distinct  from  that  of  hardware.  This  means  that  re¬ 
quirements  are  determined  specifically  for  software  and  that  the 
government  software  quality  assurance  organizations  and  procedures 
are  developed  around  ensuring  the  accomplishment  of  these  require¬ 
ments  . 

Life  cycle  model  planning  is  the  foundation  for  the  system 
design  whereby  the  best  methods  to  attain  the  end  quality  product 
are  determined.  "Best  methods"  and  "quality"  are  the  key  words. 
Quality  software  cannot  be  attained  unless  the  methods  of  imple¬ 
mentation  are  reviewed,  assessed  and  corrected  where  necessary  at 
every  step  in  each  phase  of  the  software  development  process. 

Finally,  no  software  quality  assurance  and  software  develop¬ 
ment  management  plan  will  by  themselves  cause  the  quality  of  soft¬ 
ware  to  increase.  The  most  critical  factor  is  personnel  who  have 
the  technical  expertise  to  evaluate  software  requirements,  develop 
tools  and  techniques  to  review  and  assess  compliance  at  the  various 
milestones,  and  recommend  changes  with  adequate  justification. 
Computer  technology  is  constantly  changing  and  advancing  and 
provision  must  be  made  to  update  personnel  in  the  state-of-the-art. 
Continual  training  is  essential  ,  both  for  those  personnel  who  have 
a  computer  science  background  and  those  who  do  not.  A  Training 
Course  Outline  designed  to  cover  the  major  topics  of  software 
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quality  assurance  for  Air  Force/Electronic  Systems  Division  con¬ 
tracts  has  been  developed  and  submitted  by  Systems  Architects,  Inc 

It  should  be  noted  that  planning  for  changes  and  implementa¬ 
tion  may  result  in  increased  costs  in  the  beginning.  But  as  these 
changes  become  a  matter  of  accepted  routine,  as  contractors  become 
accustomed  to  project  demands  and  begin  to  address  them  in  their 
original  plans,  and  when  personnel  become  familiar  with  new  pro¬ 
cedures,  a  decrease  in  costs  should  occur  and  an  improved  quality 
of  software  should  result. 

Based  on  the  analysis  of  current  software  quality  assurance 
practices  (Section  III),  the  evaluation  of  change  results  (Section 

IV) ,  and  the  detailed  evaluation  of  areas  for  improvement  (Section 

V)  ,  recommendations  are  provided  for  improvements  in  the  above 
mentioned  four  major  areas.  The  following  subsections  present: 

•  General  recommendations  applicable  to  all  organizations, 

•  Specific  recommendations  that  may  be  applicable  to  a 
particular  organization, 

•  Benefits  to  be  expected  from  implementation  of 
recommendations,  and 

•  Alternative  recommendations,  if  implementation  of 
specific  recommendations  is  not  immediately  feasible. 

Implementation  of  recommendations  will  lead  to  an  increase  in  the 
quality  of  software  if  the  cooperation  of  all  personnel  is  en¬ 
couraged  and  initial  periods  of  adjustment  are  allowed. 

6.2  RECOMMENDATIONS  FOR  THE  IMPROVEMENT  OF  SOFTWARE  QUALITY 
ASSURANCE  GUIDANCE  DOCUMENTS 

If  the  quality  of  software  is  to  improve,  greater  emphasis 
must  be  placed  on  software  quality  assurance  as  a  separate  disci¬ 
pline.  Quality  software  cannot  be  attained  by  following  hardware 
oriented  plans  and  procedures.  A  software  quality  assurance 
guidance  document  should  contain  the  following  elements: 
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•  Provision  for  clear  division  of  responsibilities  between 
the  Program  Office  and  the  Contract  Administration 
Office  and  within  each  organization, 

•  References  to  appropriate  standards  and  regulations, 

•  Basic  software  quality  assurance  requirements  to  be 
specified  in  contracts, 

•  Evaluation  and  corrective  action  procedures  specifically 
developed  for  software  quality  assurance,  and 

•  Provisions  for  developing  communications  methods  and 
responsibilities . 

6.2.1  General  Recommendations 

Before  any  improvements  in  software  quality  assurance 
guidance  documents  can  be  effective,  each  organization  must 
insist  on  the  following: 

(1)  Software  quality  assurance  requirements  should 
be  specified  in  request  for  proposals  and 
resulting  contracts. 

(2)  Software  quality  assurance  personnel  should 
participate  in  request  for  proposal  preparation, 
review  of  proposals  and  contract  award  negotia¬ 
tions  . 

(3)  Software  quality  assurance  should  be  included 
as  a  separate  cost  item  during  the  conceptual 
phase  to  ensure  that,  from  the  start,  it  is 
considered  as  an  important  aspect  of  the  soft¬ 
ware  development  process. 

(4)  Program  Offices  and  software  quality  assurance 
personnel  should  communicate  before  Memoranda 
of  Agreement  and  Letters  of  Delegation  are 
written. 

AFSCR  800-42,  Program  Office/AFCMD  Interface  ,  was 
issued  on  1  May  1981  and  "prescribes  the  policy  for 
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establishing  program  office/AFCMD  interface  during  the  con¬ 
ceptual  phase  and  for  providing  AFCMD  specialized  support  for 
AFSC  program  offices  throughout  the  acquisition  cycle".  This 
regulation  emphasizes  that  the  Air  Force  recognizes  that  all 
appropriate  personnel  should  be  involved  in  the  early  acqui¬ 
sition  phases  of  systems  acquisition. 

6.2.2  Specific  Recommendations  Applicable  to  the  Defense 
Logistics  Agency 

(1)  Expand  DLAM  8200.1,  Section  IX,  Part  15  into  a 
unique  software  quality  assurance  guidance 
document . 


(2)  Supplement  referenced  standards  and  regulations 
to  specifically  address  software  quality 
assurance.  For  example,  MIL-S-52779A  is 
referenced  as  a  typical  specification  applicable 
to  computer  software  data  items.  Software 
quality  assurance  personnel  are  to  determine 
that  the  requirements  of  MIL-S-52779A  are  met. 

But  there  are  no  details  provided  to  ensure 
that  the  requirements  are  adequately  met. 

(5)  The  Procedures  Evaluation  section  of  DLAM  8200.1, 
Section  IX,  Part  15  should  be  revised  to  address 
software  quality  assurance  procedures. 

(4)  The  Procedures  Review  section  of  the  above  men¬ 
tioned  document  should  be  revised  to  address 
the  adequacy  of  contractor's  software  quality 
assurance  procedures. 

6.2.3  Specific  Recommendations  Applicable  to  the  Electronic 

Systems  Division 

(1)  Supplement  AFR  800-14  with  a  unique  software 

quality  assurance  guidance  document.  Software 
is  addressed  by  this  document  but  software 
quality  assurance  is  not. 
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(2)  Incorporate  software  quality  assurance  functions 
and  "asks  into  the  supplement. 

(3)  Rev'se  referenced  standards  and  regulations  to 
specifically  address  software  quality  assurance. 
For  example,  see  6.2.2  (2). 

(4)  Supplement  AFSCR  800-42,  7a.  to  read  "in  every 
case  establish  a  formal  program  office/AFCMD 
interface  for  all  major  or  designated  systems, 
etc . " . 

6.2.4  Specific  Recommendations  Applicable  to  the  Air  Force 

Contract  Management  Division 

(1)  Expand  AFCMDR  74-1,  Chapter  15  into  a  unique 
software  quality  assurance  guidance  document. 

(2)  Revise  referenced  standards  and  regulations  in 
AFCMDR  74-1,  Chapter  15  to  specifically  address 
software  quality  assurance.  For  example,  see 
6.2.2  (2). 

(5)  Develop  a  section  on  software  quality  assurance 
for  CMSEP . 

(4)  Supplement  AFSCR  800-42  to  require  establishment 
of  a  history  or  contract  folder  starting  with 
involvement  at  the  conceptual  phase.  This  will 
establish  continuity  when  contract  responsibil¬ 
ities  are  assumed  by  an  AFPRO  or  DLA  organiza¬ 
tion  upon  contract  award. 

(5)  Encourage  wider  use  of  AFCMDR  800-1,  Contract 
Management  Engineering  and  AFCMDP  800-2 
Contract  Management  Engineering  Guide. 

6.2.5  Bene  fits 

(1)  Specification  of  software  quality  assurance  as 
a  contractual  requirement  provides  for  its  en¬ 
forcement  throughout  software  development. 
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(2)  Participation  of  software  quality  assurance 
personnel  in  early  system  acquisition  reduces 
the  possibility  of  requirements  being  omitted. 

(3)  Improved  matching  of  tasks  and  personnel  re¬ 
sources  is  ensured  through  communications  be¬ 
tween  Program  Offices  and  Contract  Administra¬ 
tion  Offices. 

(4)  Improved  schedule  performance  is  ensured  by  de¬ 
tailing  software  quality  assurance  requirements 
and  tasks  in  early  acquisition  phases. 

(5)  All  of  the  above  should  result  in  cost 
reduction. 

6.2.6  Alternative  Recommendations 

Electronic  Systems  Division's  Software  Acquisition 
Guidebooks  may  be  adapted  by  all  three  organizations.  The 
guidebooks  are  clearly  software  oriented  and  provide  de¬ 
tailed  procedures  for  all  phases  of  system  development. 

Again,  it  is  recommended  that  software  quality  assurance 
personnel  be  involved  in  early  system  acquisition  phases. 

The  Defense  Logistics  Agency's  miniaturized  version 
of  a  consolidated  manual,  Defense  In-Plant  Quality  Assurance 
Program,  combines  in  one  document  all  pertinent  manuals  re¬ 
garding  quality  assurance.  Again,  this  manual  is  hardware 
oriented.  The  same  type  of  manual  should  be  prepared  for 
software  to  include  more  detail  based  on  software  principles 
rather  than  adapting  hardware  procedures  to  software. 

The  Electronic  Systems  Division  can  prepare  the  same 
type  of  consolidated  manual  combining  AFR  800-14  with  all 
referenced  regulations  and  specifications  regarding  software. 
A  combination  of  this  manual  and  the  Software  Acquisition 
Guidebooks  would  foster  an  effective  management  program. 

The  Air  Force  Contract  Management  Division  can  prepare 
a  consolidated  manual  including  Chapter  15  of  AFCMDR  74-1, 
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AFCMDR  800-1,  AFCMDP  800-2,  other  referenced  regulations  and 
specifications  on  software,  and  CMSEP  with  the  addition  of 
questions  specifically  geared  to  software  quality  assurance. 
A  consolidated  manual  will  provide  the  software  quality 
assurance  staff  with  basic  information  and  reduce  time  spent 
in  document  search.  The  next  step  would  be  to  revise  the 
documents  to  include  more  detail  based  on  software. 

6.3  RECOMMENDATIONS  FOR  IMPROVEMENT  OF  THE  RELATION  OF  SOFTWARE 


Software  quality  assurance  is  an  integral  part  of  the  soft¬ 
ware  development  process.  This  must  be  reflected  in  the  life 
cycle  planning.  Software  quality  assurance  activities  must  be 
planned  with  the  same  precision  and  concern  as  every  other  detail 
of  software  development.  If  not,  there  is  no  reason  for  its 
existence.  Quality  does  not  just  happen.  It  is  the  result  of 
such  things  as  rigid  control,  frequent  assessment  through  reviews 
and  audits,  and  pre-planned  corrective  action  procedures. 

Specifically,  some  of  the  functions  covered  by  software 
quality  assurance  are: 

•  Ensuring  that  sound  software  development  practices  are 
followed  throughout  the  software  development  life  cycle, 

•  Establishing  software  quality  assurance  evaluation  pro¬ 
cedures  , 

•  Developing  tools  and  techniques  tailored  for  monitoring 
contractor  reviews  and  audits, 

•  Monitoring  the  review  of  all  acceptance  test  plans  and 
procedures . 

6.3.1  General  Recommendations 

Based  on  the  above  criteria,  the  following  recom¬ 
mendations  are  made  for  improving  the  relation  of  soft¬ 
ware  quality  assurance  to  software  development.  These  re¬ 
commendations  apply  to  the  Defense  Logistics  Agency,  the 


Electronic  Systems  Division  and  the  Air  Force  Contract 
Management  Division. 


(1)  Integrate  software  quality  assurance  activities 
into  every  phase  of  life  cycle  models  and 
development . 

(2)  Develop  model  documentation  standard  formats 
for  writing  documents. 

(3)  Develop  standard  checklists  specifically  for 
monitoring  software  quality  assurance  audits 
and  reviews. 

(4)  Develop  procedures  that  are  independent  of  hard¬ 
ware  oriented  procedures. 

6.3.2  Specific  Recommendation  Applicable  to  the  Defense 

Logistics  Agency 

(1)  A  major  review  of  the  life  cycle  model  should 
be  undertaken  by  the  Defense  Logistics  Agency 
to  be  consistent  with  the  one  used  within  DoD. 
This  model  is  accepted  by  industry. 

6.3.3  Benefits 

(lj  Life  cycle  models  with  software  quality  assur¬ 
ance  activities  integrated  can  serve  as  guides 
for  planning.  Requirements,  procedures,  and 
documentation  required  by  each  contract  will  be 
inserted  into  the  appropriate  areas  of  the  life 
cycle  phase.  From  this  point,  detailed  planning 
can  take  place  with  negligible  chance  of  im¬ 
portant  areas  being  omitted. 

(2)  Software  quality  assurance  personnel  may  be  re¬ 
sponsible  for  several  projects,  all  of  which  may 
be  proceeding  through  different  phases.  Fully 
developed  life  cycle  models  with  integrated 
software  quality  assurance  activities,  provide 

a  method  of  keeping  track  of  which  activities 

are  to  be  performed,  and  reports  that  are  to  be 
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written  for  each  project.  By  using  this 
tracking  system,  the  software  quality  assurance 
staff  can  plan  and  use  time  efficiently  and, 
again,  less  danger  exists  of  important  areas 
being  omitted. 

(3)  A  life  cycle  model  as  described  provides  a  sig¬ 
nificantly  practical  tool  for  training  software 
quality  assurance  personnel.  Transfer  of  know¬ 
ledge  from  the  training  setting  to  the  world  of 
everyday  activities  will  be  facilitated  by  the 
use  of  a  tool  that  depicts  standard  procedures 
in  software  development  projects. 

(4)  Since  the  system  of  controls  for  software 
quality  assurance  is  based  primarily  on  documen¬ 
tation  review,  model  documents  will  serve 

as  guides  for  documentation  preparation. 

(5)  Tools  tailor  made  for  specific  contracts  will 
address  software  quality  requirements  and  elim¬ 
inate  the  use  of  standard  hardware  quality 
procedures . 

6.3.4  Alternative  Recommendations 

AFCMD's  Contractor  Management  Self  Evaluation 
Plan  (CMSEP)  is  a  technique  that  can  be  adapted 
by  the  Defense  Logistics  Agency  and  the  Elec¬ 
tronic  Systems  Division.  This  plan  is  highly 
structured  and  provides  a  tracking  system  for 
evaluating  contractors'  fulfillment  of  contrac¬ 
tual  requirements.  Again,  include  a  section 
specifically  written  for  software  quality 
assurance . 

Electronic  Systems  Division  uses  checklists 
modeled  on  Data  Item  Descriptions  (DIDs)  and 
Standards.  Work  on  consolidation  of  all 
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checklists  has  also  begun.  These  efforts  can  be 
undertaken  by  the  Defense  Logistics  Agency  and 
the  Air  Force  Contract  Management  Division. 


6.4  RECOMMENDATIONS  FOR  IMPROVEMENT  OF  COMMUNICATION  METHODS  AND 
MODEL  DOCUMENTS 

Many  communications  fail  because  of  inadequate  planning. 

Good  planning  must  consider  the  responsibilities  of  those  who  will 
receive  the  communications  and  those  who  will  be  affected  by  it. 
While  communications  may  be  aimed  primarily  at  meeting  the  demands 
of  an  immediate  situation,  they  must  be  consistent  with  long-range 
objectives . 

The  following  guides  are  presented  to  aid  in  developing  ef¬ 
fective  communication: 

•  Determine  the  purpose  of  the  communication; 

•  Develop  a  system  of  communication  flow  to  be  followed 
during  project  implementation; 

•  Determine  appropriate  guidelines  and  regulations;  and 

•  Ensure  that  all  affected  project  personnel  are  made 
aware  of  the  communication  system. 

6.4.1  General  Recommendations 

Based  on  the  above  criteria,  the  following  recom¬ 
mendations  are  made  for  improving  communication  methods 
and  model  documents  within  and  among  organizations.  These 
recommendations  apply  to  the  Defense  Logistics  Agency,  the 
Electronics  Systems  Division  and  the  Air  Force  Contract 
Management  Division. 

(1)  Determine  formal  and  informal  methods  of  com¬ 
munication. 

(2)  For  written  communications,  develop  model 
communication  documents. 

(3)  Determine  which  communications  are  to  be  main¬ 
tained  as  records. 
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(4)  Develop  a  tracking  system  for  communication 
flow. 

(5)  Determine  the  requirements  for  frequency  of 
communicat ions . 

6.4.2  Specific  Recommendations:  None 

6.4.3  Benefits 

(1)  Model  communication  documents  will  be  uniform 
and  consistent  throughout  the  organization, 
formatted  and  prepared  for  use  so  that  informa¬ 
tion  is  brief  but  complete. 

(2)  Maintaining  records  ensures  that  all  personnel 
have  access  to  relevant  information  and  new 
personnel  can  be  updated  concerning  significant 
events  in  project  performance. 

(3)  A  tracking  system  will  ensure  that  all  personnel 
receive  communications  that  affect  their  activ¬ 
ities,  and  are  aware  of  who  is  to  receive  re¬ 
ports,  letters,  etc.  that  they,  in  turn,  are 
required  to  submit. 

(4)  Determining  the  frequency  of  communications  aids 
personnel  in  preparing  beforehand  for  confer¬ 
ences  and  meetings  and  encourages  the  mainte¬ 
nance  of  records  that  may  be  the  subject  of  the 
meeting. 

6.4.4  Alternative  Recommendations:  None 

6 . 5  RECOMMENDATIONS  FOR  IMPROVEMENT  OF  TRAINING  AND  STAFFING 

The  key  factor  in  assuring  the  quality  of  software  is  profes¬ 
sionally  trained  personnel  who  operate  as  a  separate  group  within 
an  organization.  Hardware  and  software  are  integrated;  one  cannot 
run  without  the  other.  Hardware  quality  assurance  and  software 
quality  assurance  groups  must  integrate  the  results  of  their  activ¬ 
ities  but  the  performance  of  activities  requires  the  use  of  dif¬ 
ferent  plans  and  procedures. 
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Software  literature  emphasizes  the  fact  that  future  quality 
groups  may  combine  software  quality  assurance  and  hardware 
quality  assurance  into  one  total  quality  assurance  organization. 
For  the  present,  SAI  recommends  the  formation  of  separate  groups. 
Software  quality  assurance  groups  should  operate  independently, 
develop  unique  software  quality  assurance  guidance  documents 
and  evaluation  procedures,  develop  specific  regulations  and 
standards,  and  employ  personnel  technically  on  a  par  with  their 
hardware  quality  assurance  counterparts.  When  software  quality 
assurance  is  recognized  as  equally  important  and  functions  as  a 
unique  discipline,  then  integration  can  occur. 

6.5.1  General  Recommendations 

Based  on  these  considerations,  the  Defense  Logistics 
Agency,  the  Electronic  Systems  Division  and  the  Air  Force 
Contract  Management  Division  are  encouraged  to  maintain  and 
strengthen  separate  software  quality  assurance  organizations 
To  this  end,  the  following  recommendations  are  made  for 
improvements  in  training  and  staffing. 

(1)  Provide  training  in  the  management  of  software 
projects  and  in  basic  software  engineering 
principles . 

(2)  Make  training  available  to  more  personnel. 

(3)  Develop  programs  for  certification. 

(4)  Orient  personnel  to  specific  contractor 
organizations,  procedures  and  communication 
methods . 

6.5.2  Specific  Recommendations  Applicable  to  the  Defense 

Logistics  Agency 

(1)  In  the  revision  of  DLAM  8220.4  which  DLA  is 
undertaking,  include  courses  on  integration 
of  software  quality  assurance  activities  with 
the  software  development  life  cycle. 


98 


(2)  Supplement  DLAM  8220.5,  Quality  Assurance  Intern 
Program  to  include  software  quality  assurance. 

6.5.3  Specific  Recommendations  Applicable  to  the  Electronic 

Systems  Division  and  the  Air  Force  Contract 

Management  Division 

(1)  Investigate  adoption  of  the  Defense  Logistics 
Agency's  Quality  Assurance  Certification  Program 
DLAM  8220.4,  and  Quality  Assurance  Intern  Pro¬ 
gram  DLAM  8220.5. 

(2)  Supplement  both  of  the  above  programs  to  in¬ 
clude  software  quality  assurance  if  this  has 
not  been  done . 

6.5.4  Benefits 

(1)  Training  in  the  use  of  software  design  and 
development  tools  and  techniques  will  enable 
personnel  to  perform  software  quality  assurance 
functions  as  a  group  separate  from  and  indepen¬ 
dent  of  any  other  group. 

(2)  Certification  programs  strengthen  the  profession¬ 
al  level  of  project  staff.  With  the  increase  in 
emphasis  on  software  quality  assurance,  RFPs  may 
specify  that  certified  contractor  software 
quality  assurance  professionals  must  be  assigned 
to  software  development  projects.  Government 
software  quality  assurance  personnel  should 
possess  the  same  qualifications. 

(3)  Orientation  instructions  ensure  that  personnel 
are  familiarized  with  the  organization  and  its 
procedures . 

(4)  Trained  software  quality  assurance  personnel  can 
perform  activities  with  skill  and  confidence, 
and  ensure  that  projects  are  completed  within 
time  and  cost  constraints. 
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6.5.5  Alternative  Recommendations 

Training  cannot  be  made  available  to  all  personnel 
except  over  a  long  period  of  time.  In  the  interim,  and 
where  possible,  less  experienced  personnel  can  be  afforded 
the  opportunity  to  work  on  a  daily  basis  with  experienced 
government  software  quality  assurance  managers. 

6.6  CONCLUSIONS 

Greater  quality  of  software  can  be  expected  from  software 
quality  assurance  guidance  documents  that  treat  software  quality 
assurance  as  a  unique  discipline,  software  life  cycle  development 
that  integrates  software  quality  assurance  activities,  communica¬ 
tion  methods  that  are  structured  yet  flexible,  and  personnel  who 
receive  the  training  needed  to  assist  them  in  the  performance  of 
their  work. 

Implementation  of  these  recommendations  will  provide  DLA, 
ESD  and  AFCMD  with  an  increased  probability  of  receiving  and  main 
taining  quality  software. 


APPENDIX  A 


TRAINING  COURSE  OUTLINE 
FOR  IMPROVING 

SOFTWARE  QUALITY  ASSURANCE 


METHODOLOGY  FOR  IMPLEMENTING  TRAINING  COURSE 


Overview 

This  Training  Course  Outline  was  prepared  by  Systems  Archi¬ 
tects,  Incorporated  (SAI) .  It  is  designed  to  cover  the  major 
topics  of  Software  Quality  Assurance  (SQA)  for  Air  Force/Elec¬ 
tronic  Systems  Division  contracts.  The  course  is  introductory  in 
nature,  and  upon  completion,  students  should  be  familiar  with 
terminology,  contractual  requirements,  regulations,  specifications 
and  standards  concerning  software  quality  assurance.  The  study 
should  acquire  a  basic  familiarity  with  and  appreciation  for  SQA, 
and  a  knowledge  of  where  to  turn  for  further  information. 

Suggested  Approach 

This  outline  defines  a  stand  alone  course.  The  material 
should  take  approximately  eighty  (80)  hours  to  cover.  With  the 
exception  of  Section  6.0,  sections  have  been  broken  down  to  at 
least  three  levels.  Further  subdivisions  will  occur  during  actual 
instruction. 

Prerequisites 

Students  should  have  some  familiarity  with  software  (such  as 
programming  experience)  and  some  familiarity  with  the  organization 
through  an  orientation  and,  if  possible,  some  on-the-job  training. 

Materials 

Students  should  be  provided  with  hard  copies  of  materials 
prior  to  discussion.  This  primarily  includes  software-related 
documents;  for  example,  those  documents  listed  in  Section  2.0 
(2.2).  Overhead  slides  or  handouts  which  emphasize  the  important 
points  should  be  prepared. 


A- 1 


Course  Overview 


Section  1.0  --  Software  Development  Throughout  the  Software 
Life  Cycle 

This  section  is  an  introduction  to.  the  software  life  cycle 
and  highlights  important  software  development  activities  related 
chronologically  to  the  life  cycle.  The  life  cycle  will  be  the 
central  reference  throughout  the  training  course;  it  should  be 
appropriately  emphasized. 

Section  2.0  --  Software  Quality  Assurance  Requirements 

This  section  begins  with  an  introduction  to  SQA  and  how  it 
relates  to  and  supports  software  development  throughout  the  life 
cycle.  It  introduces  important  documents,  directed  at  SQA.  Also 
introduced  are  software  reviews  and  audits,  which  form  the  back¬ 
bone  of  SQA  activities. 

Section  3.0  --  Preparation  for  Contract  Award  and  Monitoring 

This  section  covers  activities  up  to  and  including  contract 
award.  Discussions  include  the  Request  for  Proposal  and  its  com¬ 
ponents,  delegation  of  responsibilities  to  the  CAO  prior  to 
contract  award,  documents  guiding  pre-award  and  source  selection 
activities,  and  the  contract  award  activities. 

Section  4.0  --  Contract  Administration 

This  section  covers  contract  administration  activities.  It 
encompasses  a  discussion  of  documents  dealing  with  configuration 
management,  procedures  for  evaluating  progress  throughout  the 
contract,  a  discussion  of  procedures  to  report  schedule  deficien¬ 
cies  or  major  changes,  and  a  discussion  of  validation  procedures 
and  their  advantages  and  disadvantages.  Finally,  the  most  fre¬ 
quent  forms  of  documentation  should  be  highlighted  along  with  a 
discussion  of  who  is  responsible  for  their  preparation  and  who 
reviews  them. 
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Section  S.O 


SQA  Within  the  Government  Organizations 


At  this  point,  software  quality  assurance  is  discussed  more 
specifically  in  the  context  of  the  government  organization.  First 
a  brief  introduction  should  be  given  regarding  Air  Force  SQA 
responsibilities  and  how  they  relate  to  the  particular  organiza¬ 
tion.  Next  is  a  detailed  study  of  SQA  in  the  particular  organi¬ 
zation. 

Section  6.0  --  SQA  Within  the  Contractor  Organization 

Contractor  responsibilities  regarding  SQA  are  discussed. 
Activities  that  are  generally  part  of  the  contractor's  management 
plan  for  software  quality  assurance  are  covered  and  should  be 
correlated  to  the  government’s  procedures  for  monitoring  these 
plans . 

Section  7.0  --  Communications 

This  section  will  include  discussions  on  formal  and  informal 
communication  methods,  oral  and  written.  Its  purpose  will  be  to 
clearly  define  the  communication  network  and  key  points  of  refer¬ 
ence.  Also  covered  will  be  important  communications  reports,  and 
people  responsible  for  their  preparation  and  review.  The  goal  of 
this  section  is  to  familiarize  students  with  where  to  turn  for 
information,  and  to  make  them  aware  of  their  communications  re¬ 
sponsibilities.  This  includes  communications  within  the  govern¬ 
ment  organization,  among  the  government  organizations  (including 
the  Administrative  Contracting  Offices)  and  with  the  contractor. 

Section  8.0  --  Modern  Software  Engineering  Tools  and  Techniques 

This  section  describes  modern  tools  which  aid  software 
development  and  quality  assurance  throughout  the  life  cycle.  Some 
examples  are  given,  but  emphasis  should  be  placed  on  the  different 
types  of  tools  which  are  useful  in  each  phase  of  the  life  cycle, 
and  their  uses.  Some  software  development  techniques  are  included 
as  examples.  This  section  can  be  adapted  to  include  techniques 
most  often  used  in  Air  Force  software  development  programs. 
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Section  9.0  --  Workshop 

This  section  is  the  culmination  of  the  course,  and  is  in¬ 
tended  to  be  taught  in  a  workshop  format.  It  draws  upon  know¬ 
ledge  gained  in  the  first  seven  sections.  Students  will  be 
given  an  opportunity  to  develop  a  typical  SQA  program  from  the 
conceptual  phase  through  the  operational  phase.  The  SQA  pro¬ 
gram  will  be  developed  based  on  a  sample  software  development 
contract . 
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1.0  SOFTWARE  DEVELOPMENT  PLAN  THROUGHOUT  THE  SOFTWARE  LIFE 


1.1  Requirements  Analysis  Phase 

1.1.1  Program  Management  Directive  (PMD) 

1.1.2  Program  Management  Plan  (PMP) 

1.1.3  Test  and  Evaluation  Master  Plan  (TEMP) 

1.1.4  Computer  Resources  Integrated  Support  Plan  (CRISP) 

1.1.5  Systems  Specifications 

1.1.6  Systems  Requirements  Review  (SSR) 

1.1.7  Systems  Design  Review  (SDR) 

1.2  Preliminary  Design 

1.2.1  Computer  Program  Configuration  Item  (CPCI) 

1.2.2  Part  I  Computer  Program  Development  Specifications 

1.2.3  Preliminary  Design  Review  (PDR) 

1.2.4  CPCI  Test  Plan 

1.3  Detail  Design 

1.3.1  Draft  Part  II  Computer  Program  Product 
Specifications 

1.3.2  Critical  Design  Review  (CDR) 

1.3.3  Part  II  Specifications  and  Test  Procedures 

1.4  Code  and  Checkout 

1.4.1  Computer  Program  Test  and  Evaluation  (CPT  5  E) 

1.4.2  Altered  Product  Specifications 

1.4.3  Preliminary  Qualifying  Test  (PQT)  Reports 

1.5  Test  and  Integration 

1.5.1  Formal  Qualification  Test  (FQT) 

1.5.2  CPCI  Adaptation,  Installation  and  Checkout 

1.5.3  CPCI  Qualification  Final  Report 

1.5.4  Functional  Configuration  Audit  (FCA) 

1.5.5  Physical  Configuration  Audit  (PCA) 

1.5.6  Formal  Qualification  Review  (FQR) 

1.5.7  Development  Test  and  Evaluation  (DT8E) 

1.5.8  Initial  Operation  Test  and  Evaluation  (IOT  §  E) 

1.5.9  Program  Management  Responsibility  Transfer 
(PMR1 ) 
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1.6  Operation  and  Support 

1.6.1  Follow-on  Operational  Test  6  Evaluation 
(FOT  5  E) 

2.0  SOFTWARE  QUALITY  ASSURANCE  REQUIREMENTS 

2.1  Interrelation  of  Software  Development  and  Software 

Quality  Assurance 

2.1.1  How  Software  Quality  Assurance  and  Software 
Development  Activities  Integrate  Into  the 
Life  Cycle  Model 

2.1.2  Assurance  That  Software  Complies  With  Design 
and  Performance  Requirements 

2.1.3  Reduction  of  Schedule  Slippages  By  Early 
Problem  Identification  and  Isolation 

2.1.4  Systematic  Recording,  Reporting,  and  Tracking 
Software  Problems  to  Assure  That  Prompt 
Corrective  Actions  Are  Taken 


2.2  Documents  Directed  at  SQA 

2.2.1  Management  Plans 

2. 2. 1.1  AFR  800-14 

Vol.  I  -  Management  of  Computer 
Resources  in  System? 

Vol.  II-  Acquisition  and  Support 
Procedures  for  Computer 
Resources  in  Systems' 

2. 2. 1.2  AFCMDR  74-1  Procurement  Quality 
Assurance  Program 

2. 2. 1.3  AFCMDR  178-1  Contractor  Management 
System  Evaluation  Program 

2. 2. 1.4  DLAM  8200.1  Procurement  Quality 
Assurance 

2. 2. 1.5  Computer  Program  Development  Plans 

2.2.2  Software  Quality  Guidance  Documents 

2. 2. 2.1  MIL-S-52779A  Software  Quality 
Assurance  Program  Requirements 

2.2. 2. 2  AFR  74-1,  Chapter  15  "Computer 
Software  Quality  Assurance  Program" 
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2. 2. 2. 3  ESD-TR-  77 - 2 55  Software  Quality 

Assurance  ™ 

2. 2. 2. 4  RADC-TR- 77 - 369  "Factors  In  Software 
Quality" 


2.2.3  Configuration  Management  Guidance  Documents 


2. 2. 3.1  MIL-STD-483  Configuration  Manage¬ 
ment  Practices  for  Systems,  Equip¬ 
ment,  Munitions,  and  Computer 
Programs' 

2. 2. 3. 2  MIL-STD-490  Specification  Practices 

2. 2. 3. 3  ESD-TR- 77 - 2 54  An  Air  Force  Guide  to 
Computer  Program  Configuration 
Management 


2.2.4  Software  Development  Guidance  Documents 


2. 2. 4.1  MIL-STD-1679  (Navy)  Weapon  System 
Software  Development 

2. 2.4. 2  ESD-TR- 77- 1 30  Software  Development 
and  Maintenance  Facilities 


2. 2.4. 3  ESD-TR-75-85  An  Air  Force  Guide  to 

Monitoring  and  'Reporting  Software 
Development  Status 


2.3  Software  Reviews  and  Audits 


2.3.1  Systems  Requirements  Reviews  (SRR) 

2.3.2  Systems  Design  Review  (SDR) 

2.3.3  Preliminary  Design  Review  (PDR) 

2.3.4  Critical  Design  Review  (CDR) 

2.3.5  Functional  Configuration  Audit  (FCA) 

2.3.6  Physical  Configuration  Audit  (PCA) 

2.3.7  Formal  Qualification  Review  (FQR) 

2.3.8  Code  and  Design  Walkthroughs 

3.0  PREPARATION  FOR  CONTRACT  AWARD  AND  MONITORING 

3.1  Request  for  Proposal  (RFP) 

3.1.1  Statement  of  Work  (SOW) 

3.1.2  Contract  Data  Requirements  hist  (CPRL) 
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3.1.3  Write  Instruction  For  Proposal  Preparation 
(IFPP) 

3.1.4  Pre- Award  Survey 

3.2  Source  Selection 

3.2.1  Preparation  of  Evaluation  Criteria 

3.2.2  Proposal  Review 

3.2.3  Assess  Proposals 

3.2.4  Contract  Award  Negotiations 

3.3  Division  of  Responsibilities 

3.3.1  Memoranda  of  Agreement  (MOA) 

3.3.2  Letter  of  Delegation  (LOD) 

3.3.3  Letter  of  Instruction  (LOI) 

3.3.4  PMP,  CRISP 

3.4  Management  Guidance  Documents 

3.4.1  AFR  800-14  Management  of  Computer  Resources 
in  Systems 


3.4.2  AFCMDR  74-1  Procurement  Quality  Assurance 
Program  ” 


3.4.3  AFCMDR  178-1  Contractor  Management  System 
Evaluation  Program 


3.4.4  DLAM  8200.1  (AFR  74-15)  Procurement  Quality 
Assurance 

4.0  CONTRACT  ADMINISTRATION 

4.1  Configuration  Management  Guidance  Documents 

4.1.1  DoD-STD-480A  Configuration  Control -Engineering 
Changes,  Deviations^  and  waivers 

4.1.2  MIL-STD-483  Configuration  Management  Practices 
for  Systems,  Equipment,  Munitions  and  Computer 
Programs 


4.1.3  ESD-TR- 77- 2 54  An  Air  Force  Guide  to  Computer 
Program  Configuration  Management 


4.2  Evaluation  Procedures 


4.2.1  Checklists 

4.2.2  Testing 
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4.2.3  Baselines 

4.2.4  Reviews  and  Audits  (Milestones) 

4.3  Validation 

4.3.1  Prepare  and  Evaluate  Standards  Guiding 
Preparation  of  Software  Documentation,  Code, 
and  Verification 

4.3.2  Monitor  Software  Design  for  Compliance  With 
Design  and  Performance  Requirements  and 
Adequacy  of  Methods  Used 

4.3.3  Plan  Milestones  at  Which  to  Evaluate  Software 
Verification  Results 

4.4  Design,  Test,  and  Code  Control 

4.4.1  Design  Control  (Conformance  to  Requirements) 

4.4.2  Test  Control 

4.4.3  Code  Control  (Conformance  to  Standards) 

4.5  Corrective  Action  Procedures 

4.5.1  Software  Trouble  Reports  (STR) 

4.5.2  Deficiency  Reports 

4.5.3  Engineering  Change  Proposal  (ECP) 

4.5.4  Briefings  and  Meetings  (e.g.,  Conf igurat ion 
Control  Board) 

4.6  Documentation 

4.6.1  Types  of  Documentation  and  Who  Prepares  Them 

4.6.2  Placement  of  Documentation  in  Life  Cycle 

4.6.3  Descriptions  of  Standard  Documents  (e.g.,  Desi 
Specifications,  CPDP,  CRISP,  Manuals,  Reports, 
etc . ) 

5. 0  SQA  WITHIN  THE  GOVERNMENT  ORGANIZATIONS 

5.1  Air  Force-Wide  SQA  Responsibilities 

5.1.1  Setting  of  SQA  Objectives  and  Priorities 

5.1.2  Establishing  and  Managing  Enforcement  of 
Regulations  and  Standards 

5.1.3  Ongoing  Research  and  Development  of  SQA 
Measuring  Techniques 

5.1.4  Standardization  and  Automation  of  SQA 
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5.1.5  Provide  for  Government  Review  at  Contractor, 
Subcontractor,  or  Vendor  Facilities  to 
Determine  Conformance  to  SQA  Requirements 

5.2  Electronic  Systems  Division  Software  Quality 
Assurance 

5.2.1  Air  Force  Software  Quality  Assurance  Programs 

5.2.2  Program  Office  SQA 

5. 2. 2.1  Review  Contractor's  QA  Program 

5. 2. 2. 2  Perform  Document  Reviews 

5. 2. 2. 3  Review  Computer  Program  Development 
Specifications  CBS's) 

5. 2. 2. 4  Review  Computer  Program  Product 
Specifications  (C5's) 

5. 2. 2. 5  Review  Test  Plans 

5. 2. 2.6  Maintain  Government  Records 

5.2.3  Computer  Systems  Evaluation  Panel  (CSEP) 

5.3  Air  Force  Contract  Management  Division  (AFCMD)  SQA 

5.3.1  Air  Force  SQA  Programs 

5.3.2  Directorate  of  Engineering  (Office  of  Primary 
Responsibility  for  Embedded  Computer  Resources) 

5.3.3  Directorate  of  Quality  Assurance  (Office  of 
Collateral  Responsibility  for  Software) 

5.3.4  Detachment  QA  Divisions  (Responsible  for 
Surveillance  of  QA  Aspects  of  the  Software 
Development  Cycle) 

5.4  Defense  Logistics  Agency  (DLA)  SQA 

5.4.1  Air  Force  SQA  Programs 

5.4.2  Basic  QA  Program  Within  DLA 

5.4. 2.1  Concepts  §  Planning 

5.4. 2. 2  Review  of  Procedures 

5.4. 2. 3  Evaluation  of  Procedures 

5. 4. 2. 4  Product  Verification  Inspection 

5. 4. 2. 5  Corrective  Action 


CONTRACTOR  SQA  PROGRAMS 


6.1  Pre-award  Surveys  and  Post-award  Conferences 
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6.2  Configuration  Management 

6.3  Reviews  and  Audits 


6.4  Test  Plans  a. d  evaluation  Reports 

6.5  Corrective  Actions 

6.6  Software  Documentation 

6.7  Library  Control 

6.8  Subcontractor  Control 
7. 0  COMMUNICATIONS 

7.1  Communication  Needs 

7.1.1  Between  Contract  Administration  Office 
(CAO)  and  Program  Office  (PO) 

7.1.2  Interorganization 

7.1.3  Within  Organization 

7.1.4  Between  Government  Organizations  and  Contractor 

7.2  Principles  of  Communications 

7.2.1  Define  Content  of  Communication 

7.2.2  Define  Communication  Medium  (Report, 

Telephone,  Memo,  etc.) 

7.2.3  Determine  Frequency  of  Communications 

7.2.4  Define  Sender,  Recipient,  Carbon  Copies, 
and  Channel  Which  Communication  Should 
Follow 

7.2.5  Provide  Guidelines  and  Regulations  Out¬ 
lining  the  Communication  Network  and 
Communication  Flow 

7.2.6  Identify  Key  Points  of  Contact 

7.3  Formal  Written  Communications 

7.3.1  Guidelines  and  Regulations 

7.3.2  Reports 

7.3.3  Letters 

7.3.4  Memoranda 
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7.4  Formal  Oral  Communications 


7.4.1  Meetings,  Panel  Discussions 

7.4.2  Training  Sessions,  Classes,  Orientations 

7.4.3  Briefings,  Speeches 

7.5  Informal  Written  Communications 

7.5.1  Notes,  Informal  Memoranda 

7.5.2  Suggestions,  Grievances 

7.6  Informal  Oral  Communications 

7.6.1  Face-to-face  Discussions 

7.6.2  Telephoning 

7.6.3  Work  Groups 

8.0  MODERN  SOFTWARE  ENGINEERING  TOOLS  AND  TECHNIQUES 

8.1  Automated  Test  Tools  Throughout  The  Software 

Development  Process 

8.1.1  Requirement  Analysis/Preliminary  Design 

8. 1.1.1  Automated  Design  Tools  (Methodology 
for  Stating  Requirements  for  Design). 
Examples  of  this  include  CADSAT 
(USAF) ,  DQM  (Hughes)  and  MEDL-R 
(Martin  Marietta) 

8. 1.1. 2  Automated  Simulation  Tools  (Model 
the  Hardware/Software  of  the  System 
to  Study  its  Characteristics).  An 
example  is  AISIM  (Hughes) 

8.1.2  Detail  Design,  Coding  and  Checkout 

8. 1.2.1  Code  Analysis  Tools  (To  Find  Syntax 
Errors  and  Error-prone  Constructions) 

8. 1.2. 2  Structure  Analysis  (To  Look  for 
Structural  Flaws) 

8. 1.2. 3  Module  Interface  Checks  (Find 
Inconsistencies  in  Declaration  of 
Data  Structures  and  Check  Module 
Linkage)  ARGUS  MICRO  (Boeing)  is 
an  Example  of  This 

8. 1.2. 4  Check  Sequence  of  Events  (e.g., 

Order  of  Input/Output  Sequences, 
etc . ) 
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8.1.3  Test  and  Verification 

8. 1.3.1  Monitor  Run-Time  Behavior  (To 
Collect  Execution  Statistics) 

8. 1.3. 2  Automated  Test  Case  Generation 
(To  Provide  Teste.rs  with  Best 
Test  Cases  for  Each  System) 

8.1.4  Performance  Factors  (Prior  to  Installation) 

8. 1.4.1  Program  Restructuring  Tools  (Assist 
Program  Reorganization  for 
Optimization) 

8. 1.4. 2  Validation  of  Parallel  Operations 
(To  Evaluate  Efficiency  of  Parallel 
Processing  Schedule) 

8.1.5  Maintenance  and  Documentation 

8. 1.5.1  Documentation  Generation  (Highlights 
Code  and  Structure  Information 
Pertinent  to  Documentation) 

Examples  Include  TRANSFOR  (Boeing) , 
GIRAFF  (USAF) ,  and  COMMON  (Naval 

R  8  D  Center) 

8.1. 5.2  Evaluation  of  Modification  (To- 
Predict  Effects  of  Proposed  Changes) 

8.1.6  Software  Quality  Validation  Tools  (Compare 

Program  Attributes  to  a  Set  of  Desirable 

Characteristic  Attributes)  An  Example  is 

METRICS  (GE) 

8.2  Software  Development  Techniques 

8.2.1  Structured  Programming 

8. 2.1.1  Top-Down  Design 

8. 2. 1.2  Structured  Design 

8.2.2  Control  Structures 

8. 2. 2.1  Sequence  Structure 

8. 2. 2. 2  Selection  Structure 

8. 2. 2. 3  Iteration  Structure 

8. 2. 2. 4  Exit  Structure 

8.2.3  Techniques 

8. 2.3.1  Flowcharts 
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8. 2. 3. 2  Pseudocode 

8. 2. 3. 3  Schematic  Logic 

9.0  WORKSHOP 

9.1  Requirements  Identification  (For  Each  Life  Cycle 
Phase) 

9.1.1  Develop  Life  Cycle  Model 

9.1.2  Determine  Baseline  Documentation 

9.1.3  Integrate  Appropriate  SQA  Activities 

9.1.4  Determine  Applicable  SQA  Tools 

9.2  SQA  Tool  Development 

9.2.1  Assign  Students  to  Working  Groups 

9.2.2  Assigr  Each  Group  to  a  Life  Cycle  Phase 

9.2.3  Create  SQA  Tools  Based  on  Phase  Requirements 

9.2.4  Critique 

9. 2. 4.1  Group  Exchange 

9. 2. 4. 2  Class  Discussion 

9.2.5  Improve  Tools 

9.3  SQA  Tool  Application 

9.3.1  Re-assign  Students  to  Working  Groups 

9.3.2  Use  Tools  to  Evaluate  Contractor 
Documentation 

9.3.3  Evaluate  Tool  Usefulness 

9.3.4  Improve  Tools 

9. 3. 4.1  Group  Exchange 

9. 3.4. 2  Class  Discussion 


MISSION 

of 

Rome  Air  Development  Center 


RAVC  plan*  and  tx.ec utes  research,  development,  tut  and 
selected  acquisition  programs  in  support  of  Command,  Compute 
Communications  and  Intelligence.  (C3I)  activities .  Technical 
and  engineering  support  usi thin  areas  technical  competence 
is  provided  to  ESP  Program  Offices  IPOs)  and  other  ESO 
elements .  The  principal  technical  mission  areas  are 
communications ,  electromagnetic  guidance  and  control,  *«*- 
veillanci  of  ground  and  aerospace  objects,  intelligence,  data 
collection  and  handling,  information  system  technology, 
ionospheric  propagation,  solid  state  sciences,  micromve 
physics  and  electronic  reliability,  maintainability  and 
compatibility. 


